[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260927 23:29:14.187388 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:46049
I20260927 23:29:14.188643 396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:29:14.189625 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:14.210705 402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:14.210757 403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:14.211248 405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:14.213150 396 server_base.cc:1034] running on GCE node
I20260927 23:29:15.425202 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:15.425493 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:15.425698 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551755425678 us; error 0 us; skew 500 ppm
I20260927 23:29:15.426627 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:15.432008 396 webserver.cc:460] Webserver started at http://127.0.99.62:40019/ using document root <none> and password file <none>
I20260927 23:29:15.432861 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:15.433091 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:15.433533 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:15.440315 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/master-0-root/instance:
uuid: "aac04c8bfc4a4e88a80128e74708ff59"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:15.448818 396 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260927 23:29:15.454614 411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:15.455785 396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 23:29:15.456144 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/master-0-root
uuid: "aac04c8bfc4a4e88a80128e74708ff59"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:15.456519 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:15.499215 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:15.500701 396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:29:15.501190 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:15.579882 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:46049
I20260927 23:29:15.580024 462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:46049 every 8 connection(s)
I20260927 23:29:15.587620 463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:15.612221 463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: Bootstrap starting.
I20260927 23:29:15.618412 463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:15.620139 463 log.cc:824] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: Log is configured to *not* fsync() on all Append() calls
I20260927 23:29:15.624963 463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: No bootstrap required, opened a new log
I20260927 23:29:15.648507 463 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER }
I20260927 23:29:15.649097 463 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:15.649287 463 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac04c8bfc4a4e88a80128e74708ff59, State: Initialized, Role: FOLLOWER
I20260927 23:29:15.650538 463 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER }
I20260927 23:29:15.650995 463 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:29:15.651211 463 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:29:15.651494 463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:15.655390 463 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER }
I20260927 23:29:15.656056 463 leader_election.cc:302] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac04c8bfc4a4e88a80128e74708ff59; no voters:
I20260927 23:29:15.658093 463 leader_election.cc:288] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:29:15.658478 466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:15.661490 466 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 1 LEADER]: Becoming Leader. State: Replica: aac04c8bfc4a4e88a80128e74708ff59, State: Running, Role: LEADER
I20260927 23:29:15.662480 466 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER }
I20260927 23:29:15.663406 463 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:29:15.677448 467 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac04c8bfc4a4e88a80128e74708ff59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER } }
I20260927 23:29:15.677480 468 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac04c8bfc4a4e88a80128e74708ff59. Latest consensus state: current_term: 1 leader_uuid: "aac04c8bfc4a4e88a80128e74708ff59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac04c8bfc4a4e88a80128e74708ff59" member_type: VOTER } }
I20260927 23:29:15.678229 467 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [sys.catalog]: This master's current role is: LEADER
I20260927 23:29:15.678570 468 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [sys.catalog]: This master's current role is: LEADER
I20260927 23:29:15.681994 476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:29:15.694927 476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:29:15.697410 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 23:29:15.713827 476 catalog_manager.cc:1347] Generated new cluster ID: acbdc2260bb4483bac1fafc509779c61
I20260927 23:29:15.714134 476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:29:15.735733 476 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:29:15.737680 476 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:29:15.751227 476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: Generated new TSK 0
I20260927 23:29:15.752271 476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:29:15.765108 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:15.775453 485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:15.777561 486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:15.780238 488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:15.781885 396 server_base.cc:1034] running on GCE node
I20260927 23:29:15.782675 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:15.782897 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:15.783128 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551755783111 us; error 0 us; skew 500 ppm
I20260927 23:29:15.783740 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:15.788406 396 webserver.cc:460] Webserver started at http://127.0.99.1:36397/ using document root <none> and password file <none>
I20260927 23:29:15.788941 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:15.789149 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:15.789424 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:15.790736 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-0-root/instance:
uuid: "842a1c42cccc4b9390e0a4464a696512"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:15.796765 396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 23:29:15.801740 493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:15.802645 396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 23:29:15.802934 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-0-root
uuid: "842a1c42cccc4b9390e0a4464a696512"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:15.803227 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:15.819199 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:15.820752 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:15.823503 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:15.828392 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:15.828605 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:15.828794 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:15.828975 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:15.937438 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:44343
I20260927 23:29:15.937482 555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:44343 every 8 connection(s)
I20260927 23:29:15.946102 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:15.956773 560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:15.959340 561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:15.976682 563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:15.978648 396 server_base.cc:1034] running on GCE node
I20260927 23:29:15.979653 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:15.979923 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:15.980114 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551755980100 us; error 0 us; skew 500 ppm
I20260927 23:29:15.980916 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:15.981616 556 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46049
I20260927 23:29:15.982112 556 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:15.983163 556 heartbeater.cc:511] Master 127.0.99.62:46049 requested a full tablet report, sending...
I20260927 23:29:15.984233 396 webserver.cc:460] Webserver started at http://127.0.99.2:39549/ using document root <none> and password file <none>
I20260927 23:29:15.984912 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:15.985143 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:15.985466 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:15.986253 428 ts_manager.cc:194] Registered new tserver with Master: 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343)
I20260927 23:29:15.986972 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-1-root/instance:
uuid: "794c8bb85c71461e9bb257c6afaec6eb"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:15.988659 428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60672
I20260927 23:29:15.994160 396 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 23:29:15.999235 568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.000151 396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:29:16.000492 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-1-root
uuid: "794c8bb85c71461e9bb257c6afaec6eb"
format_stamp: "Formatted at 2026-09-27 23:29:15 on dist-test-slave-zkzt"
I20260927 23:29:16.000829 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:16.023408 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:16.025218 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:16.027796 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:16.032056 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:16.032343 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.032624 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:16.032831 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.157913 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:43691
I20260927 23:29:16.158042 630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:43691 every 8 connection(s)
I20260927 23:29:16.165817 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:16.176169 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:16.185920 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:16.189019 631 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46049
I20260927 23:29:16.189456 631 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:16.190413 631 heartbeater.cc:511] Master 127.0.99.62:46049 requested a full tablet report, sending...
I20260927 23:29:16.192992 428 ts_manager.cc:194] Registered new tserver with Master: 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
W20260927 23:29:16.194633 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:16.195293 396 server_base.cc:1034] running on GCE node
I20260927 23:29:16.196323 428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60674
I20260927 23:29:16.197175 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:16.197918 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:16.198133 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551756198119 us; error 0 us; skew 500 ppm
I20260927 23:29:16.198868 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:16.201836 396 webserver.cc:460] Webserver started at http://127.0.99.3:40319/ using document root <none> and password file <none>
I20260927 23:29:16.202575 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:16.202806 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:16.203116 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:16.204622 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-2-root/instance:
uuid: "7798fd3291924994932b048007322c25"
format_stamp: "Formatted at 2026-09-27 23:29:16 on dist-test-slave-zkzt"
I20260927 23:29:16.210781 396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 23:29:16.215190 642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.216086 396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:29:16.216439 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-2-root
uuid: "7798fd3291924994932b048007322c25"
format_stamp: "Formatted at 2026-09-27 23:29:16 on dist-test-slave-zkzt"
I20260927 23:29:16.217202 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790551754024304-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:16.257772 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:16.259407 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:16.261945 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:16.265101 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:16.265384 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.265650 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:16.265843 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:16.367362 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:36007
I20260927 23:29:16.367439 704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:36007 every 8 connection(s)
I20260927 23:29:16.388350 705 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46049
I20260927 23:29:16.388758 705 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:16.389730 705 heartbeater.cc:511] Master 127.0.99.62:46049 requested a full tablet report, sending...
I20260927 23:29:16.392306 428 ts_manager.cc:194] Registered new tserver with Master: 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:16.392617 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015930208s
I20260927 23:29:16.395503 428 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60676
I20260927 23:29:16.399555 396 test_util.cc:268] Using random seed: 1365533609
I20260927 23:29:16.473266 428 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60680:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:29:16.478691 428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:29:16.611748 669 tablet_service.cc:1467] Processing CreateTablet for tablet ba86a414c8484f1282985adfd9d4f0dd (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 23:29:16.613584 669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba86a414c8484f1282985adfd9d4f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.609812 670 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb9a657b844a0ab14c93f43a140b5f (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 23:29:16.619104 665 tablet_service.cc:1467] Processing CreateTablet for tablet 26ae0bb2648b49cc96a48f44c5c446be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 23:29:16.622745 665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ae0bb2648b49cc96a48f44c5c446be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.627247 664 tablet_service.cc:1467] Processing CreateTablet for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 23:29:16.630082 667 tablet_service.cc:1467] Processing CreateTablet for tablet 94efcd0874294a779f8c486e9e28ad75 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 23:29:16.634106 663 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbcdea459084f5796ecec2f8db827be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 23:29:16.633769 666 tablet_service.cc:1467] Processing CreateTablet for tablet 34ead357ebc64eec911e793d35f180b1 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 23:29:16.636958 670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb9a657b844a0ab14c93f43a140b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.638831 668 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd5edb0ddf4a649ebd29ca2df70e23 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 23:29:16.647181 664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.643164 662 tablet_service.cc:1467] Processing CreateTablet for tablet 83126b384e0d437994105143a69490c9 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 23:29:16.648849 595 tablet_service.cc:1467] Processing CreateTablet for tablet ba86a414c8484f1282985adfd9d4f0dd (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 23:29:16.649879 663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbcdea459084f5796ecec2f8db827be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.647186 596 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb9a657b844a0ab14c93f43a140b5f (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 23:29:16.652316 594 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd5edb0ddf4a649ebd29ca2df70e23 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 23:29:16.651049 593 tablet_service.cc:1467] Processing CreateTablet for tablet 94efcd0874294a779f8c486e9e28ad75 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 23:29:16.660847 595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba86a414c8484f1282985adfd9d4f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.663188 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dd5edb0ddf4a649ebd29ca2df70e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.656598 590 tablet_service.cc:1467] Processing CreateTablet for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 23:29:16.667156 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83126b384e0d437994105143a69490c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.667032 588 tablet_service.cc:1467] Processing CreateTablet for tablet 83126b384e0d437994105143a69490c9 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 23:29:16.668277 593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94efcd0874294a779f8c486e9e28ad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.663638 589 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbcdea459084f5796ecec2f8db827be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 23:29:16.669237 590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.666239 591 tablet_service.cc:1467] Processing CreateTablet for tablet 26ae0bb2648b49cc96a48f44c5c446be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 23:29:16.669360 592 tablet_service.cc:1467] Processing CreateTablet for tablet 34ead357ebc64eec911e793d35f180b1 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 23:29:16.686506 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dd5edb0ddf4a649ebd29ca2df70e23. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:29:16.694062 475 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:29:16.695602 596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb9a657b844a0ab14c93f43a140b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.697717 592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ead357ebc64eec911e793d35f180b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.696163 666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ead357ebc64eec911e793d35f180b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.700174 591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ae0bb2648b49cc96a48f44c5c446be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.702672 588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83126b384e0d437994105143a69490c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.707252 667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94efcd0874294a779f8c486e9e28ad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.716251 589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbcdea459084f5796ecec2f8db827be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.731770 518 tablet_service.cc:1467] Processing CreateTablet for tablet 94efcd0874294a779f8c486e9e28ad75 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 23:29:16.733629 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94efcd0874294a779f8c486e9e28ad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.739984 517 tablet_service.cc:1467] Processing CreateTablet for tablet 34ead357ebc64eec911e793d35f180b1 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 23:29:16.742130 517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ead357ebc64eec911e793d35f180b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.718765 521 tablet_service.cc:1467] Processing CreateTablet for tablet 8beb9a657b844a0ab14c93f43a140b5f (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 23:29:16.747745 521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8beb9a657b844a0ab14c93f43a140b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.738147 520 tablet_service.cc:1467] Processing CreateTablet for tablet ba86a414c8484f1282985adfd9d4f0dd (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 23:29:16.750819 520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba86a414c8484f1282985adfd9d4f0dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.729781 519 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd5edb0ddf4a649ebd29ca2df70e23 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 23:29:16.754704 519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dd5edb0ddf4a649ebd29ca2df70e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.763394 516 tablet_service.cc:1467] Processing CreateTablet for tablet 26ae0bb2648b49cc96a48f44c5c446be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 23:29:16.787577 516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ae0bb2648b49cc96a48f44c5c446be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.797179 515 tablet_service.cc:1467] Processing CreateTablet for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 23:29:16.826465 515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fafcbb858a14e1198b5e3eb02e1c9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.826395 514 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbcdea459084f5796ecec2f8db827be (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 23:29:16.829139 514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbcdea459084f5796ecec2f8db827be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.833191 513 tablet_service.cc:1467] Processing CreateTablet for tablet 83126b384e0d437994105143a69490c9 (DEFAULT_TABLE table=test-workload [id=9c10e0f448be4b0ca004a073444d97bb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 23:29:16.847923 513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83126b384e0d437994105143a69490c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:16.970544 724 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:16.994160 556 heartbeater.cc:503] Master 127.0.99.62:46049 was elected leader, sending a full tablet report...
I20260927 23:29:17.006371 723 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.024017 724 tablet_bootstrap.cc:654] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.033226 725 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.050079 723 tablet_bootstrap.cc:654] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.060799 725 tablet_bootstrap.cc:654] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.068864 723 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.069303 724 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.069350 723 ts_tablet_manager.cc:1397] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.007s
I20260927 23:29:17.069854 724 ts_tablet_manager.cc:1397] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.107s user 0.012s sys 0.007s
I20260927 23:29:17.071354 725 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.071762 725 ts_tablet_manager.cc:1397] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.004s
I20260927 23:29:17.072690 723 raft_consensus.cc:348] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.072940 724 raft_consensus.cc:348] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.073683 723 raft_consensus.cc:374] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.073709 724 raft_consensus.cc:374] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.074257 723 raft_consensus.cc:729] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.074249 724 raft_consensus.cc:729] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.074501 725 raft_consensus.cc:348] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.075467 725 raft_consensus.cc:374] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.075896 725 raft_consensus.cc:729] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.075284 723 consensus_queue.cc:260] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.075764 724 consensus_queue.cc:260] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.077494 725 consensus_queue.cc:260] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.078243 705 heartbeater.cc:503] Master 127.0.99.62:46049 was elected leader, sending a full tablet report...
I20260927 23:29:17.078718 724 ts_tablet_manager.cc:1428] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260927 23:29:17.079928 724 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.088379 724 tablet_bootstrap.cc:654] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.091562 631 heartbeater.cc:503] Master 127.0.99.62:46049 was elected leader, sending a full tablet report...
I20260927 23:29:17.092293 725 ts_tablet_manager.cc:1428] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.020s user 0.010s sys 0.002s
I20260927 23:29:17.093313 725 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.097867 723 ts_tablet_manager.cc:1428] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.028s user 0.011s sys 0.001s
I20260927 23:29:17.101610 723 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.101773 725 tablet_bootstrap.cc:654] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.111426 729 raft_consensus.cc:482] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.112097 729 raft_consensus.cc:504] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.115985 729 leader_election.cc:288] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.116227 723 tablet_bootstrap.cc:654] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
W20260927 23:29:17.133044 706 tablet.cc:2369] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:29:17.135623 723 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.136183 723 ts_tablet_manager.cc:1397] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.015s
I20260927 23:29:17.139482 723 raft_consensus.cc:348] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.140303 723 raft_consensus.cc:374] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.140645 723 raft_consensus.cc:729] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.141444 723 consensus_queue.cc:260] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.143909 723 ts_tablet_manager.cc:1428] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260927 23:29:17.144903 723 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.151539 723 tablet_bootstrap.cc:654] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.160168 724 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.161089 724 ts_tablet_manager.cc:1397] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.004s
I20260927 23:29:17.171962 723 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.172605 723 ts_tablet_manager.cc:1397] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.004s
I20260927 23:29:17.179472 724 raft_consensus.cc:348] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.180729 724 raft_consensus.cc:374] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.181130 724 raft_consensus.cc:729] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.181892 724 consensus_queue.cc:260] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.179641 723 raft_consensus.cc:348] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.185719 724 ts_tablet_manager.cc:1428] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.024s user 0.004s sys 0.007s
I20260927 23:29:17.184485 723 raft_consensus.cc:374] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.186424 723 raft_consensus.cc:729] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.187247 723 consensus_queue.cc:260] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.188565 725 tablet_bootstrap.cc:492] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.189172 725 ts_tablet_manager.cc:1397] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.096s user 0.027s sys 0.047s
I20260927 23:29:17.189697 724 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.195040 723 ts_tablet_manager.cc:1428] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.022s user 0.005s sys 0.003s
I20260927 23:29:17.196501 723 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.197721 725 raft_consensus.cc:348] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.198580 725 raft_consensus.cc:374] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.199071 725 raft_consensus.cc:729] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.199827 725 consensus_queue.cc:260] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.200452 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:17.201695 531 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 0.
I20260927 23:29:17.202437 725 ts_tablet_manager.cc:1428] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260927 23:29:17.203706 645 leader_election.cc:302] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.206593 725 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.206626 729 raft_consensus.cc:2781] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.207163 729 raft_consensus.cc:482] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.207952 729 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.207098 724 tablet_bootstrap.cc:654] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.218298 723 tablet_bootstrap.cc:654] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.229315 725 tablet_bootstrap.cc:654] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.235622 738 raft_consensus.cc:482] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.233944 729 raft_consensus.cc:504] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:17.241621 729 leader_election.cc:288] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.236614 738 raft_consensus.cc:504] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.240935 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:17.243426 531 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.244882 724 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.246941 724 ts_tablet_manager.cc:1397] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.004s
I20260927 23:29:17.254050 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.255856 606 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 0.
I20260927 23:29:17.255728 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.257228 605 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.258940 738 leader_election.cc:288] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.260205 731 raft_consensus.cc:482] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.259465 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
W20260927 23:29:17.268659 645 leader_election.cc:341] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.269619 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.270277 606 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 0.
I20260927 23:29:17.261910 731 raft_consensus.cc:504] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.272596 643 leader_election.cc:302] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:17.273902 738 raft_consensus.cc:2781] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.274232 738 raft_consensus.cc:482] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.274559 738 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.281775 731 leader_election.cc:288] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.302704 724 raft_consensus.cc:348] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.303527 724 raft_consensus.cc:374] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.303818 724 raft_consensus.cc:729] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.304828 724 consensus_queue.cc:260] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.307212 724 ts_tablet_manager.cc:1428] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.052s user 0.001s sys 0.006s
I20260927 23:29:17.311264 723 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.311728 723 ts_tablet_manager.cc:1397] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.116s user 0.014s sys 0.013s
I20260927 23:29:17.313606 724 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.317824 725 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.318468 725 ts_tablet_manager.cc:1397] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.112s user 0.009s sys 0.016s
I20260927 23:29:17.323952 723 raft_consensus.cc:348] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.324764 723 raft_consensus.cc:374] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.325042 723 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.325160 531 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 1.
I20260927 23:29:17.325696 723 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.326409 645 leader_election.cc:302] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.326212 738 raft_consensus.cc:504] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.328616 738 leader_election.cc:288] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.329146 738 raft_consensus.cc:2781] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.329995 738 raft_consensus.cc:686] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:17.330106 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:17.330654 605 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 1.
I20260927 23:29:17.331097 724 tablet_bootstrap.cc:654] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.330878 738 consensus_queue.cc:237] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
W20260927 23:29:17.332779 645 leader_election.cc:341] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.332520 725 raft_consensus.cc:348] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.333297 725 raft_consensus.cc:374] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.334009 725 raft_consensus.cc:729] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.334044 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.334681 605 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.336882 723 ts_tablet_manager.cc:1428] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.025s user 0.004s sys 0.003s
I20260927 23:29:17.334848 725 consensus_queue.cc:260] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.339799 723 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.359484 723 tablet_bootstrap.cc:654] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.365016 725 ts_tablet_manager.cc:1428] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.045s user 0.005s sys 0.003s
I20260927 23:29:17.375550 725 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.375355 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.377058 680 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 0.
I20260927 23:29:17.380872 495 leader_election.cc:302] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.381953 731 raft_consensus.cc:2781] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.382498 731 raft_consensus.cc:482] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.382920 731 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.387578 724 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.389201 724 ts_tablet_manager.cc:1397] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.006s
I20260927 23:29:17.399870 730 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.399770 724 raft_consensus.cc:348] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.400519 730 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.404896 730 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.401229 724 raft_consensus.cc:374] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.409356 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
W20260927 23:29:17.412544 494 leader_election.cc:341] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.414199 725 tablet_bootstrap.cc:654] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.411159 724 raft_consensus.cc:729] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.417373 724 consensus_queue.cc:260] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.420498 724 ts_tablet_manager.cc:1428] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.031s user 0.004s sys 0.004s
I20260927 23:29:17.421406 724 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.421653 723 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.422152 723 ts_tablet_manager.cc:1397] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.012s
I20260927 23:29:17.421558 425 catalog_manager.cc:5495] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:17.424496 731 raft_consensus.cc:504] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.424923 723 raft_consensus.cc:348] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.425652 723 raft_consensus.cc:374] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.425944 723 raft_consensus.cc:729] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.426800 605 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 1.
I20260927 23:29:17.428931 643 leader_election.cc:302] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:17.431579 738 raft_consensus.cc:2781] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.432067 738 raft_consensus.cc:686] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:17.433379 738 consensus_queue.cc:237] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.434350 731 leader_election.cc:288] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.427249 723 consensus_queue.cc:260] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.435091 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.436316 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:17.438155 680 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.450752 723 ts_tablet_manager.cc:1428] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
W20260927 23:29:17.444705 494 leader_election.cc:341] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.455358 723 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.460520 724 tablet_bootstrap.cc:654] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.476825 723 tablet_bootstrap.cc:654] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.478783 725 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.480329 725 ts_tablet_manager.cc:1397] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.105s user 0.015s sys 0.013s
I20260927 23:29:17.487859 725 raft_consensus.cc:348] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.488677 725 raft_consensus.cc:374] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.490661 725 raft_consensus.cc:729] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.495435 725 consensus_queue.cc:260] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.497678 725 ts_tablet_manager.cc:1428] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260927 23:29:17.508949 724 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.509545 724 ts_tablet_manager.cc:1397] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.020s
I20260927 23:29:17.502619 725 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.509825 731 raft_consensus.cc:482] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.511101 731 raft_consensus.cc:504] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.515168 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.515909 605 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 0.
I20260927 23:29:17.517010 494 leader_election.cc:302] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.513819 731 leader_election.cc:288] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.522747 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.531556 680 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:17.532109 731 raft_consensus.cc:2781] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.532691 731 raft_consensus.cc:482] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.533089 731 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.531644 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
W20260927 23:29:17.537734 495 leader_election.cc:341] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7798fd3291924994932b048007322c25 (127.0.99.3:36007): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.538757 495 leader_election.cc:302] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters: 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:17.546471 724 raft_consensus.cc:348] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.553097 756 raft_consensus.cc:2781] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.554155 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:17.554952 724 raft_consensus.cc:374] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.555430 724 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
W20260927 23:29:17.556376 571 leader_election.cc:341] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.560030 723 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.560567 723 ts_tablet_manager.cc:1397] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.105s user 0.019s sys 0.035s
I20260927 23:29:17.560762 756 raft_consensus.cc:686] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:17.561414 730 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.562167 730 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.562309 756 consensus_queue.cc:237] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.565115 730 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.566484 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.566524 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
W20260927 23:29:17.569532 570 leader_election.cc:341] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7798fd3291924994932b048007322c25 (127.0.99.3:36007): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.566555 724 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
W20260927 23:29:17.573402 571 leader_election.cc:341] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.574414 678 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 0.
I20260927 23:29:17.574638 571 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb; no voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:17.575448 730 raft_consensus.cc:2726] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:17.575887 724 ts_tablet_manager.cc:1428] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.066s user 0.007s sys 0.002s
I20260927 23:29:17.575052 723 raft_consensus.cc:348] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.576713 723 raft_consensus.cc:374] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.577179 724 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.577090 723 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.577533 570 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:17.579732 730 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.580125 730 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.578444 723 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.580499 730 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.586045 724 tablet_bootstrap.cc:654] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.600191 425 catalog_manager.cc:5495] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:17.605603 724 tablet_bootstrap.cc:492] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.606180 724 ts_tablet_manager.cc:1397] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.005s
I20260927 23:29:17.588069 725 tablet_bootstrap.cc:654] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.616062 724 raft_consensus.cc:348] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.617005 758 raft_consensus.cc:482] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.620558 758 raft_consensus.cc:504] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.624238 758 leader_election.cc:288] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.625537 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.622984 724 raft_consensus.cc:374] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.627049 724 raft_consensus.cc:729] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.629969 724 consensus_queue.cc:260] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.628364 730 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.632681 731 raft_consensus.cc:504] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.633641 730 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.634588 731 leader_election.cc:288] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.633694 425 catalog_manager.cc:5495] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:17.628321 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.635262 678 raft_consensus.cc:2454] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 0.
I20260927 23:29:17.634433 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:17.636029 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
W20260927 23:29:17.630976 494 leader_election.cc:341] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:29:17.637141 571 leader_election.cc:341] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.638389 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.638602 495 leader_election.cc:302] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters: 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:17.638933 605 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.639652 731 raft_consensus.cc:2781] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.639988 731 raft_consensus.cc:482] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.640271 731 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.643138 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:17.643968 678 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.652247 723 ts_tablet_manager.cc:1428] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.091s user 0.019s sys 0.023s
I20260927 23:29:17.654629 680 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.658346 725 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.658882 725 ts_tablet_manager.cc:1397] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.156s user 0.023s sys 0.009s
I20260927 23:29:17.659706 723 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.661023 725 raft_consensus.cc:348] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.661588 725 raft_consensus.cc:374] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.661862 725 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.662503 725 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.664605 725 ts_tablet_manager.cc:1428] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 23:29:17.665608 725 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.668884 723 tablet_bootstrap.cc:654] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.668948 724 ts_tablet_manager.cc:1428] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260927 23:29:17.671190 724 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.678836 724 tablet_bootstrap.cc:654] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.689714 723 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.690440 723 ts_tablet_manager.cc:1397] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.019s
I20260927 23:29:17.691567 725 tablet_bootstrap.cc:654] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.693828 723 raft_consensus.cc:348] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.694713 723 raft_consensus.cc:374] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.695077 723 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.696113 723 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
W20260927 23:29:17.698243 475 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:29:17.699879 723 ts_tablet_manager.cc:1428] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20260927 23:29:17.700977 723 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.701398 724 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.701987 724 ts_tablet_manager.cc:1397] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.009s
I20260927 23:29:17.708243 725 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.708133 724 raft_consensus.cc:348] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.708899 724 raft_consensus.cc:374] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.708997 725 ts_tablet_manager.cc:1397] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.008s
I20260927 23:29:17.709201 724 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.710064 724 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.712103 725 raft_consensus.cc:348] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.712610 724 ts_tablet_manager.cc:1428] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260927 23:29:17.712846 725 raft_consensus.cc:374] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.713151 725 raft_consensus.cc:729] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.713599 724 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.713798 725 consensus_queue.cc:260] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.716305 725 ts_tablet_manager.cc:1428] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:29:17.717342 725 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.720579 723 tablet_bootstrap.cc:654] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.721549 731 raft_consensus.cc:504] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.722035 678 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 1.
I20260927 23:29:17.722957 680 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:17.724852 731 leader_election.cc:288] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.725747 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.729295 724 tablet_bootstrap.cc:654] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.729750 725 tablet_bootstrap.cc:654] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.734697 495 leader_election.cc:302] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.736096 731 raft_consensus.cc:2781] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.736701 731 raft_consensus.cc:686] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:17.739913 570 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:17.741046 730 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.741503 730 raft_consensus.cc:686] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:17.742111 725 tablet_bootstrap.cc:492] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.742578 725 ts_tablet_manager.cc:1397] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.003s
I20260927 23:29:17.742300 730 consensus_queue.cc:237] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.745420 725 raft_consensus.cc:348] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.721690 605 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:17.746277 725 raft_consensus.cc:374] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.746757 725 raft_consensus.cc:729] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.747386 725 consensus_queue.cc:260] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
W20260927 23:29:17.748088 494 leader_election.cc:341] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.749696 738 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:17.750239 725 ts_tablet_manager.cc:1428] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 23:29:17.751217 725 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.753509 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:17.737767 731 consensus_queue.cc:237] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.758342 725 tablet_bootstrap.cc:654] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.754118 680 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.780334 725 tablet_bootstrap.cc:492] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.780932 725 ts_tablet_manager.cc:1397] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.005s
I20260927 23:29:17.784618 725 raft_consensus.cc:348] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.785418 725 raft_consensus.cc:374] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.785688 725 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.786543 725 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.789615 725 ts_tablet_manager.cc:1428] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260927 23:29:17.790725 725 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: Bootstrap starting.
I20260927 23:29:17.797077 725 tablet_bootstrap.cc:654] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.808323 758 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.808770 758 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.809511 725 tablet_bootstrap.cc:492] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: No bootstrap required, opened a new log
I20260927 23:29:17.809937 725 ts_tablet_manager.cc:1397] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.009s
I20260927 23:29:17.810793 758 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.812291 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.813438 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.814009 605 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 0.
I20260927 23:29:17.815227 725 raft_consensus.cc:348] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.815958 725 raft_consensus.cc:374] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.816211 725 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Initialized, Role: FOLLOWER
W20260927 23:29:17.816498 495 leader_election.cc:341] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7798fd3291924994932b048007322c25 (127.0.99.3:36007): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.816826 725 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.817310 494 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters: 7798fd3291924994932b048007322c25
I20260927 23:29:17.818120 758 raft_consensus.cc:2781] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.818473 758 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.818813 758 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.819684 428 catalog_manager.cc:5495] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:17.825160 758 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.826058 738 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:17.828022 725 ts_tablet_manager.cc:1428] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: Time spent starting tablet: real 0.018s user 0.000s sys 0.008s
I20260927 23:29:17.829463 758 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.829911 530 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.831796 680 raft_consensus.cc:2454] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:17.834702 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.835330 606 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.838999 723 tablet_bootstrap.cc:492] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.839589 723 ts_tablet_manager.cc:1397] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.139s user 0.017s sys 0.041s
I20260927 23:29:17.842165 495 leader_election.cc:302] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters: 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:17.850384 738 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.850907 738 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.853502 738 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.854223 731 raft_consensus.cc:2781] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.854033 529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:17.854545 724 tablet_bootstrap.cc:492] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.854727 731 raft_consensus.cc:686] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:17.854831 529 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 0.
I20260927 23:29:17.855029 724 ts_tablet_manager.cc:1397] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.142s user 0.026s sys 0.033s
I20260927 23:29:17.855648 731 consensus_queue.cc:237] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.856236 645 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.856225 604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:17.858225 604 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 0.
I20260927 23:29:17.859335 738 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.859766 738 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.859827 724 raft_consensus.cc:348] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.860273 738 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.860574 724 raft_consensus.cc:374] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.860896 724 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.857962 723 raft_consensus.cc:348] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.862080 723 raft_consensus.cc:374] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.862388 723 raft_consensus.cc:729] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.863044 723 consensus_queue.cc:260] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.863240 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:17.863476 427 catalog_manager.cc:5495] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 1 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
W20260927 23:29:17.865034 495 leader_election.cc:341] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7798fd3291924994932b048007322c25 (127.0.99.3:36007): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:17.867962 724 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.869971 724 ts_tablet_manager.cc:1428] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20260927 23:29:17.870870 724 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: Bootstrap starting.
I20260927 23:29:17.875782 723 ts_tablet_manager.cc:1428] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260927 23:29:17.876653 723 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: Bootstrap starting.
I20260927 23:29:17.877068 730 raft_consensus.cc:482] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:17.877688 730 raft_consensus.cc:504] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.881352 529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:17.882154 529 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 0.
I20260927 23:29:17.882983 730 leader_election.cc:288] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:17.883482 571 leader_election.cc:302] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:17.884534 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:17.885314 680 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 0.
I20260927 23:29:17.885736 723 tablet_bootstrap.cc:654] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.886059 730 raft_consensus.cc:2781] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:17.886601 730 raft_consensus.cc:482] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:17.887028 730 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.897377 724 tablet_bootstrap.cc:654] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:17.900403 428 catalog_manager.cc:5495] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:17.911931 724 tablet_bootstrap.cc:492] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: No bootstrap required, opened a new log
I20260927 23:29:17.912358 724 ts_tablet_manager.cc:1397] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.006s
I20260927 23:29:17.918084 723 tablet_bootstrap.cc:492] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: No bootstrap required, opened a new log
I20260927 23:29:17.918565 723 ts_tablet_manager.cc:1397] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.000s
I20260927 23:29:17.922106 724 raft_consensus.cc:348] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.922976 724 raft_consensus.cc:374] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.923287 724 raft_consensus.cc:729] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.924053 724 consensus_queue.cc:260] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.926162 738 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.927023 606 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:17.928102 738 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:17.923339 723 raft_consensus.cc:348] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.929832 738 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:17.930296 723 raft_consensus.cc:374] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:17.930687 723 raft_consensus.cc:729] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Initialized, Role: FOLLOWER
I20260927 23:29:17.931490 723 consensus_queue.cc:260] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.932564 724 ts_tablet_manager.cc:1428] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20260927 23:29:17.933842 723 ts_tablet_manager.cc:1428] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260927 23:29:17.935314 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:17.935530 494 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters: 7798fd3291924994932b048007322c25
I20260927 23:29:17.935899 605 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.937070 731 raft_consensus.cc:2781] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:17.936975 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:17.937619 530 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:17.938153 731 raft_consensus.cc:686] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:17.939440 731 consensus_queue.cc:237] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:17.942277 738 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:17.962136 425 catalog_manager.cc:5495] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:17.992107 731 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.002525 606 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.029570 530 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 1.
I20260927 23:29:18.030968 645 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:18.029618 730 raft_consensus.cc:504] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.036532 730 leader_election.cc:288] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.037575 738 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:18.037853 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:18.038136 738 raft_consensus.cc:686] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:18.038371 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.038713 530 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.039068 680 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.039243 738 consensus_queue.cc:237] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.051625 428 catalog_manager.cc:5495] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:18.133671 530 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 1.
I20260927 23:29:18.135151 571 leader_election.cc:302] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:18.135449 758 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.136377 605 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 1.
I20260927 23:29:18.137341 680 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 1.
I20260927 23:29:18.138039 730 raft_consensus.cc:2781] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:18.139065 730 raft_consensus.cc:686] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:18.139866 730 consensus_queue.cc:237] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.150106 428 catalog_manager.cc:5495] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 1 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:18.163684 530 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.181325 731 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:18.194051 731 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.233798 758 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:18.234472 758 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.235008 730 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:18.237185 730 raft_consensus.cc:482] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:18.237666 730 raft_consensus.cc:504] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.238372 758 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.239857 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:18.240160 730 leader_election.cc:288] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.241562 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:18.243515 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:18.243955 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:18.244202 678 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 1.
I20260927 23:29:18.245009 680 raft_consensus.cc:2379] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 794c8bb85c71461e9bb257c6afaec6eb in current term 1: Already voted for candidate 842a1c42cccc4b9390e0a4464a696512 in this term.
I20260927 23:29:18.246146 570 leader_election.cc:302] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb; no voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:18.247078 730 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.247625 495 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters: 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:18.249279 758 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:29:18.249727 758 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 794c8bb85c71461e9bb257c6afaec6eb)
I20260927 23:29:18.255242 758 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.263579 796 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:18.270458 756 consensus_queue.cc:1035] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.338850 730 raft_consensus.cc:2726] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:18.338883 758 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.340163 530 raft_consensus.cc:1226] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Rejecting Update request from peer 794c8bb85c71461e9bb257c6afaec6eb for earlier term 1. Current term is 2. Ops: []
I20260927 23:29:18.341117 758 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.342841 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:18.344969 730 consensus_queue.cc:1046] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.345708 730 raft_consensus.cc:3032] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:18.345635 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.346089 730 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:18.346436 758 consensus_queue.cc:1035] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.347353 730 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.348456 495 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 842a1c42cccc4b9390e0a4464a696512; no voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:18.348738 730 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.350019 758 raft_consensus.cc:2726] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:29:18.360666 738 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:18.361106 738 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.363029 738 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:18.364449 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:18.364828 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:18.365626 605 raft_consensus.cc:2379] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7798fd3291924994932b048007322c25 in current term 1: Already voted for candidate 842a1c42cccc4b9390e0a4464a696512 in this term.
I20260927 23:29:18.366508 643 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7798fd3291924994932b048007322c25; no voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:18.367252 738 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:18.409633 758 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.424985 782 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.438448 782 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.443624 738 raft_consensus.cc:2726] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:18.446473 756 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:18.514472 680 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f"
dest_uuid: "7798fd3291924994932b048007322c25"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:48210
I20260927 23:29:18.515166 680 raft_consensus.cc:595] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:18.520905 680 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd"
dest_uuid: "7798fd3291924994932b048007322c25"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:48210
I20260927 23:29:18.521425 680 raft_consensus.cc:595] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:18.529933 680 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "7798fd3291924994932b048007322c25"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:48210
I20260927 23:29:18.530530 680 raft_consensus.cc:595] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:18.546388 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:18.547051 530 raft_consensus.cc:595] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:18.552510 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:18.553125 530 raft_consensus.cc:595] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:18.558408 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:18.558998 530 raft_consensus.cc:595] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:18.578156 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83126b384e0d437994105143a69490c9"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:58192
I20260927 23:29:18.578764 606 raft_consensus.cc:595] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:18.579063 606 raft_consensus.cc:603] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:29:18.594348 756 raft_consensus.cc:979] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.594823 756 raft_consensus.cc:1067] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.596474 606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:18.597043 606 raft_consensus.cc:482] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.597394 606 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.626771 758 raft_consensus.cc:979] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.627264 756 raft_consensus.cc:1067] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.629252 605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:18.629809 605 raft_consensus.cc:482] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.630156 605 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.641121 606 raft_consensus.cc:504] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.644284 606 leader_election.cc:288] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.645382 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.645633 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:18.645982 678 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.646229 530 raft_consensus.cc:3032] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:18.646588 530 raft_consensus.cc:729] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:18.647297 530 consensus_queue.cc:260] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.648350 530 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.678699 730 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.692322 730 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:18.705825 530 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:18.705873 605 raft_consensus.cc:504] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.707288 571 leader_election.cc:302] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:18.708148 605 leader_election.cc:288] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.709532 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:18.710109 530 raft_consensus.cc:3032] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:18.710409 530 raft_consensus.cc:729] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:18.710762 678 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:18.711089 530 consensus_queue.cc:260] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.712333 530 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.713503 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.722636 678 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.728715 730 raft_consensus.cc:2781] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:18.729233 730 raft_consensus.cc:686] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:18.730017 730 consensus_queue.cc:237] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.737360 427 catalog_manager.cc:5495] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 1 to 2, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 2 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:18.797909 678 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:18.797921 530 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:18.799412 570 leader_election.cc:302] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:18.800871 730 raft_consensus.cc:2781] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:18.801468 730 raft_consensus.cc:686] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:18.802779 730 consensus_queue.cc:237] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.811182 427 catalog_manager.cc:5495] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 1 to 2, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 2 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:18.818698 782 raft_consensus.cc:979] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.819191 782 raft_consensus.cc:1067] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.820796 530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40880
I20260927 23:29:18.821538 530 raft_consensus.cc:482] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.821964 530 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.878162 530 raft_consensus.cc:504] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:18.880611 530 leader_election.cc:288] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Requested vote from peers 7798fd3291924994932b048007322c25 (127.0.99.3:36007), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:18.882200 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.882539 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:18.882896 678 raft_consensus.cc:3032] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:18.883103 605 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.883285 678 raft_consensus.cc:729] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:18.884168 678 consensus_queue.cc:260] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:18.885423 678 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.891497 778 raft_consensus.cc:979] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.891975 782 raft_consensus.cc:1067] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.893939 530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40880
I20260927 23:29:18.894474 530 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.894752 530 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.906862 778 raft_consensus.cc:979] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.907249 731 raft_consensus.cc:979] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.907303 778 raft_consensus.cc:1067] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:18.907831 756 raft_consensus.cc:1067] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:18.909633 531 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40880
I20260927 23:29:18.909715 606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:18.910099 531 raft_consensus.cc:482] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.910385 606 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:18.910584 531 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.910801 606 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.956063 678 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:18.957573 495 leader_election.cc:302] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:18.959039 530 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.962121 604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:18.962873 604 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.962888 756 raft_consensus.cc:2781] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:18.961162 605 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:18.963853 678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:18.964486 678 raft_consensus.cc:3032] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:18.964819 678 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:18.965046 756 raft_consensus.cc:686] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:18.965647 530 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:18.965703 678 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:18.965934 756 consensus_queue.cc:237] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:18.967120 678 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:18.977589 427 catalog_manager.cc:5495] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 1 to 2, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:19.034829 606 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.034809 531 raft_consensus.cc:504] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.036928 606 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:19.038358 531 leader_election.cc:288] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:19.040457 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:19.040779 605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:19.041180 680 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:19.041332 605 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:19.042013 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:19.034775 678 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:19.042742 531 raft_consensus.cc:3032] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:19.043104 531 raft_consensus.cc:729] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:19.043009 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:19.043447 495 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:19.043766 679 raft_consensus.cc:3032] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:19.044150 679 raft_consensus.cc:729] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:19.044425 756 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:19.045324 756 raft_consensus.cc:686] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:19.044164 531 consensus_queue.cc:260] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.046376 531 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:19.046500 756 consensus_queue.cc:237] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.045007 679 consensus_queue.cc:260] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.047966 679 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:19.054694 427 catalog_manager.cc:5495] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 1 to 2, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:19.111318 680 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:19.111650 679 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:19.111295 604 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:19.111295 531 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:19.113996 570 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:19.114885 730 raft_consensus.cc:2781] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:19.115355 730 raft_consensus.cc:686] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:19.115176 495 leader_election.cc:302] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:19.116261 730 consensus_queue.cc:237] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.116943 756 raft_consensus.cc:2781] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:19.117456 756 raft_consensus.cc:686] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:19.118403 756 consensus_queue.cc:237] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.124507 427 catalog_manager.cc:5495] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 1 to 2, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 2 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:19.125702 428 catalog_manager.cc:5495] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 1 to 2, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:19.197510 605 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:19.198601 606 raft_consensus.cc:1226] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Rejecting Update request from peer 7798fd3291924994932b048007322c25 for earlier term 1. Current term is 2. Ops: []
I20260927 23:29:19.208387 679 raft_consensus.cc:1261] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.209532 730 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.218585 531 raft_consensus.cc:1261] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.220263 796 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.373054 531 raft_consensus.cc:1261] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.374400 821 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.385095 679 raft_consensus.cc:1261] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.386718 730 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.441807 606 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.443012 756 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.452104 679 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.454568 731 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.470193 679 raft_consensus.cc:1261] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.471490 731 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.480036 606 raft_consensus.cc:1261] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.481249 731 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.606150 531 raft_consensus.cc:1261] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.607295 796 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.615805 679 raft_consensus.cc:1261] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.617331 730 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.627627 679 raft_consensus.cc:1261] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.628747 731 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.637384 606 raft_consensus.cc:1261] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:19.639180 756 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:19.690928 730 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:19.691411 730 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.693058 730 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:19.693969 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:19.693980 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:19.695399 571 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb; no voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:19.696125 730 raft_consensus.cc:2726] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:29:19.767529 778 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 794c8bb85c71461e9bb257c6afaec6eb)
I20260927 23:29:19.767916 778 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.769505 778 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:19.770116 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512" is_pre_election: true
I20260927 23:29:19.770545 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:19.771021 531 raft_consensus.cc:2379] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7798fd3291924994932b048007322c25 in current term 2: Already voted for candidate 842a1c42cccc4b9390e0a4464a696512 in this term.
I20260927 23:29:19.771291 606 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 2.
I20260927 23:29:19.772248 643 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:19.772953 778 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:29:19.773296 778 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 794c8bb85c71461e9bb257c6afaec6eb)
I20260927 23:29:19.773667 778 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:19.812023 778 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.813742 778 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:19.814680 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:19.814901 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:19.815644 531 raft_consensus.cc:2379] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7798fd3291924994932b048007322c25 in current term 2: Already voted for candidate 842a1c42cccc4b9390e0a4464a696512 in this term.
I20260927 23:29:19.914458 606 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 2.
I20260927 23:29:19.915850 643 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters: 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:19.916790 778 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:19.917719 778 raft_consensus.cc:686] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:19.918569 778 consensus_queue.cc:237] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:19.925284 427 catalog_manager.cc:5495] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 1 to 2, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:20.215253 756 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:20.215785 756 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:20.217473 756 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:20.218286 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb" is_pre_election: true
I20260927 23:29:20.218578 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25" is_pre_election: true
I20260927 23:29:20.218941 606 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 2.
I20260927 23:29:20.219911 494 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:20.220564 756 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:29:20.220957 756 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:20.221283 756 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:20.256433 756 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:20.258394 756 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:20.259930 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:20.260690 606 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:20.262465 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:20.347376 606 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 3.
I20260927 23:29:20.348991 494 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters: 7798fd3291924994932b048007322c25
I20260927 23:29:20.349776 756 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:20.350298 756 raft_consensus.cc:686] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:20.351328 756 consensus_queue.cc:237] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:20.358075 427 catalog_manager.cc:5495] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 2 to 3, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:20.366293 531 raft_consensus.cc:1226] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Rejecting Update request from peer 7798fd3291924994932b048007322c25 for earlier term 2. Current term is 3. Ops: []
I20260927 23:29:20.367605 778 consensus_queue.cc:1046] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:20.368278 778 consensus_queue.cc:1225] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Peer 842a1c42cccc4b9390e0a4464a696512 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260927 23:29:20.368657 834 raft_consensus.cc:3032] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:20.368963 834 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:20.369570 834 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:20.370543 834 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:20.733022 606 raft_consensus.cc:1261] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:29:20.734810 756 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:20.751187 679 raft_consensus.cc:1261] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 23:29:20.751596 679 pending_rounds.cc:76] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Aborting all ops after (but not including) 1
I20260927 23:29:20.751868 679 pending_rounds.cc:97] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 23:29:20.752158 679 raft_consensus.cc:2866] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:29:20.753672 756 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:21.609737 396 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260927 23:29:21.610538 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:21.627718 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:21.628309 606 raft_consensus.cc:595] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:21.641134 531 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83126b384e0d437994105143a69490c9"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:50612
I20260927 23:29:21.641669 531 raft_consensus.cc:595] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:21.643883 531 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:50612
I20260927 23:29:21.644469 531 raft_consensus.cc:595] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:21.645401 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260927 23:29:21.645840 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:21.760470 845 raft_consensus.cc:979] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.760926 846 raft_consensus.cc:1067] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Signalling peer 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.762511 679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83126b384e0d437994105143a69490c9"
dest_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:48194
I20260927 23:29:21.763025 679 raft_consensus.cc:482] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:21.763391 679 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:21.769454 679 raft_consensus.cc:504] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.771142 679 leader_election.cc:288] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 4 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:21.771983 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:21.772111 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83126b384e0d437994105143a69490c9" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:21.772578 531 raft_consensus.cc:3032] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:29:21.772733 606 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:21.772969 531 raft_consensus.cc:729] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:21.773717 531 consensus_queue.cc:260] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.774883 531 raft_consensus.cc:3037] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:21.777801 606 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 4.
I20260927 23:29:21.778801 643 leader_election.cc:302] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:21.779918 531 raft_consensus.cc:2454] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 4.
I20260927 23:29:21.780296 850 raft_consensus.cc:2781] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:29:21.781303 850 raft_consensus.cc:686] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 4 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:21.782089 850 consensus_queue.cc:237] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.788349 428 catalog_manager.cc:5495] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 3 to 4, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:21.791756 854 raft_consensus.cc:979] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.792188 730 raft_consensus.cc:1067] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Signalling peer 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.793991 679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:48200
I20260927 23:29:21.795100 679 raft_consensus.cc:482] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:21.795478 679 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:21.801316 679 raft_consensus.cc:504] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.803057 679 leader_election.cc:288] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:21.803910 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:21.804106 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:21.804414 531 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:21.804709 606 raft_consensus.cc:3032] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:21.805042 606 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:21.805612 606 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.806607 606 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:21.809371 531 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 2.
I20260927 23:29:21.810289 645 leader_election.cc:302] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:21.811038 850 raft_consensus.cc:2781] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:21.811487 850 raft_consensus.cc:686] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:21.812276 606 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 2.
I20260927 23:29:21.812270 850 consensus_queue.cc:237] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.820555 428 catalog_manager.cc:5495] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 1 to 2, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:21.963621 845 raft_consensus.cc:979] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.964110 846 raft_consensus.cc:1067] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Signalling peer 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:21.965566 679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:48194
I20260927 23:29:21.966024 679 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:21.966297 679 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:21.972555 679 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.974108 679 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:21.974998 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:21.975142 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:21.975560 531 raft_consensus.cc:3032] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:21.975668 606 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:21.975880 531 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:21.976588 531 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.977748 531 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:21.980968 606 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:21.981910 643 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:21.982730 850 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:21.983174 850 raft_consensus.cc:686] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:21.983538 531 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:21.983844 850 consensus_queue.cc:237] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:21.989545 428 catalog_manager.cc:5495] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 2 to 3, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:22.254154 606 raft_consensus.cc:1261] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:29:22.255376 850 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:22.263952 531 raft_consensus.cc:1261] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:29:22.266019 851 consensus_queue.cc:1035] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:29:22.279626 606 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:22.280790 851 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:22.289678 531 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:22.291144 862 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:22.443420 606 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:22.444491 850 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:22.454586 531 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:22.456205 851 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:22.646803 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:22.651466 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:22.651787 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:23.652668 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:23.658628 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:23.659077 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:24.660200 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:24.665933 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:24.666435 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:25.667266 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:25.671697 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:25.672084 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:26.672892 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:26.677597 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:26.677943 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:27.678754 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:27.683648 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:27.683974 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:28.684618 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:28.689049 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:28.689381 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:29.690174 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:29.694694 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:29.695008 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:30.695758 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:30.700791 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:30.701175 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:31.727353 396 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260927 23:29:31.731478 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:31.732167 530 raft_consensus.cc:595] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:31.736639 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:31.737136 530 raft_consensus.cc:595] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:31.741328 530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:40902
I20260927 23:29:31.741734 530 raft_consensus.cc:595] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:31.746191 679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "7798fd3291924994932b048007322c25"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:48210
I20260927 23:29:31.746734 679 raft_consensus.cc:595] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:31.758167 679 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "7798fd3291924994932b048007322c25"
mode: GRACEFUL
new_leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:48210
I20260927 23:29:31.758750 679 raft_consensus.cc:595] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Received request to transfer leadership to TS 794c8bb85c71461e9bb257c6afaec6eb
I20260927 23:29:31.836298 850 raft_consensus.cc:979] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.836784 850 raft_consensus.cc:1067] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.838410 605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58182
I20260927 23:29:31.838914 605 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:31.839201 605 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:31.844559 605 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.846881 605 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 4 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:31.847645 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:31.848204 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:31.848385 530 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:31.848713 679 raft_consensus.cc:3032] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:29:31.849013 679 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:31.849779 679 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.851187 679 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:31.854947 530 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 4.
I20260927 23:29:31.856243 571 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:31.857142 878 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:29:31.857409 679 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 4.
I20260927 23:29:31.857545 878 raft_consensus.cc:686] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:31.858579 878 consensus_queue.cc:237] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.865767 428 catalog_manager.cc:5495] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 3 to 4, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 4 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:31.901108 876 raft_consensus.cc:979] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.901705 885 raft_consensus.cc:1067] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.903131 605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:31.903635 605 raft_consensus.cc:482] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:31.903956 605 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.908347 605 raft_consensus.cc:504] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:31.909855 605 leader_election.cc:288] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:31.910620 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:31.910920 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8beb9a657b844a0ab14c93f43a140b5f" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:31.911166 530 raft_consensus.cc:3032] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:31.911413 679 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.911440 530 raft_consensus.cc:729] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:31.912168 530 consensus_queue.cc:260] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:31.913265 530 raft_consensus.cc:3037] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.917275 679 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:31.918526 570 leader_election.cc:302] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:31.919158 530 raft_consensus.cc:2454] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:31.919369 878 raft_consensus.cc:2781] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:31.919930 878 raft_consensus.cc:686] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:31.920715 878 consensus_queue.cc:237] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } }
I20260927 23:29:31.927627 428 catalog_manager.cc:5495] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 2 to 3, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 3 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:31.929328 876 raft_consensus.cc:979] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.929872 885 raft_consensus.cc:1067] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.931303 605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:31.931856 605 raft_consensus.cc:482] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:31.932190 605 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:31.937629 879 raft_consensus.cc:979] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.937669 605 raft_consensus.cc:504] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.938169 850 raft_consensus.cc:1067] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:31.939523 605 leader_election.cc:288] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:31.939733 606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58182
I20260927 23:29:31.940188 606 raft_consensus.cc:482] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:31.940181 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:31.940696 606 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.940529 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dd5edb0ddf4a649ebd29ca2df70e23" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:31.940796 530 raft_consensus.cc:3032] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:31.941124 530 raft_consensus.cc:729] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:31.941124 679 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:31.941869 530 consensus_queue.cc:260] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.943053 530 raft_consensus.cc:3037] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:31.945243 679 raft_consensus.cc:2454] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:31.946048 570 leader_election.cc:302] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:31.946794 878 raft_consensus.cc:2781] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:31.947238 878 raft_consensus.cc:686] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:31.948043 878 consensus_queue.cc:237] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.946926 606 raft_consensus.cc:504] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.948638 530 raft_consensus.cc:2454] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 2.
I20260927 23:29:31.950471 606 leader_election.cc:288] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:31.951221 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:31.951359 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:31.951908 679 raft_consensus.cc:3032] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:31.952004 531 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.952258 679 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:31.953299 679 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.954303 679 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:31.957701 531 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:31.957710 428 catalog_manager.cc:5495] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 1 to 2, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 2 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:31.958770 571 leader_election.cc:302] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 794c8bb85c71461e9bb257c6afaec6eb, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:31.960086 679 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:31.960544 878 raft_consensus.cc:2781] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:31.961058 878 raft_consensus.cc:686] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:31.961777 878 consensus_queue.cc:237] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:31.970353 428 catalog_manager.cc:5495] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 2 to 3, leader changed from 7798fd3291924994932b048007322c25 (127.0.99.3) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 3 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:32.152333 876 raft_consensus.cc:979] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: : Instructing follower 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:32.152894 885 raft_consensus.cc:1067] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Signalling peer 794c8bb85c71461e9bb257c6afaec6eb to start an election
I20260927 23:29:32.154345 606 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
from {username='slave'} at 127.0.0.1:58186
I20260927 23:29:32.154917 606 raft_consensus.cc:482] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:32.155205 606 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:32.159633 606 raft_consensus.cc:504] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:32.161176 606 leader_election.cc:288] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:32.161955 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:32.162192 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba86a414c8484f1282985adfd9d4f0dd" candidate_uuid: "794c8bb85c71461e9bb257c6afaec6eb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:32.162695 531 raft_consensus.cc:3032] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:32.162777 679 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:32.162997 531 raft_consensus.cc:729] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:32.163679 531 consensus_queue.cc:260] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:32.164709 531 raft_consensus.cc:3037] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:32.167279 679 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:32.168253 570 leader_election.cc:302] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 794c8bb85c71461e9bb257c6afaec6eb; no voters:
I20260927 23:29:32.168874 531 raft_consensus.cc:2454] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 794c8bb85c71461e9bb257c6afaec6eb in term 3.
I20260927 23:29:32.169298 873 raft_consensus.cc:2781] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:32.169785 873 raft_consensus.cc:686] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Becoming Leader. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:32.170639 873 consensus_queue.cc:237] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:32.177659 428 catalog_manager.cc:5495] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb reported cstate change: term changed from 2 to 3, leader changed from 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1) to 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2). New cstate: current_term: 3 leader_uuid: "794c8bb85c71461e9bb257c6afaec6eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:32.329200 679 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:29:32.330428 873 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:29:32.340821 531 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:29:32.342902 873 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:29:32.409418 531 raft_consensus.cc:1261] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:32.410573 873 consensus_queue.cc:1035] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:32.419391 679 raft_consensus.cc:1261] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:32.420712 878 consensus_queue.cc:1035] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:32.422187 531 raft_consensus.cc:1261] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.423669 878 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:32.431193 679 raft_consensus.cc:1261] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.433317 873 consensus_queue.cc:1035] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:32.485780 531 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.487032 873 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:32.493592 679 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.495047 878 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:32.597599 679 raft_consensus.cc:1261] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.598853 878 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:32.609133 531 raft_consensus.cc:1261] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 794c8bb85c71461e9bb257c6afaec6eb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:32.610373 892 consensus_queue.cc:1035] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:34.794389 396 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260927 23:29:34.795419 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:34.803895 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:34.804668 606 raft_consensus.cc:595] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:34.807413 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:34.807943 606 raft_consensus.cc:595] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Received request to transfer leadership to TS 7798fd3291924994932b048007322c25
I20260927 23:29:34.810125 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:34.810561 606 raft_consensus.cc:595] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:34.812911 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:34.813450 606 raft_consensus.cc:595] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:34.815697 606 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
mode: GRACEFUL
new_leader_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:56296
I20260927 23:29:34.816156 606 raft_consensus.cc:595] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 LEADER]: Received request to transfer leadership to TS 842a1c42cccc4b9390e0a4464a696512
I20260927 23:29:34.817131 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:29:34.817610 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:34.864019 904 raft_consensus.cc:979] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:34.864660 873 raft_consensus.cc:1067] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Signalling peer 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:34.866580 679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be"
dest_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:48200
I20260927 23:29:34.867107 679 raft_consensus.cc:482] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:34.867436 679 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:34.872699 679 raft_consensus.cc:504] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:34.875041 679 leader_election.cc:288] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:34.875916 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:34.876155 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ae0bb2648b49cc96a48f44c5c446be" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:34.876621 531 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:34.876816 606 raft_consensus.cc:3032] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:34.877274 606 raft_consensus.cc:729] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:34.878386 606 consensus_queue.cc:260] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:34.879805 606 raft_consensus.cc:3037] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:34.882123 531 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:34.883277 645 leader_election.cc:302] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:34.884029 850 raft_consensus.cc:2781] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:34.885217 850 raft_consensus.cc:686] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:34.885340 606 raft_consensus.cc:2454] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:34.885985 850 consensus_queue.cc:237] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:34.893400 425 catalog_manager.cc:5495] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 2 to 3, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:34.995187 904 raft_consensus.cc:979] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:34.995636 873 raft_consensus.cc:1067] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:34.997069 531 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40888
I20260927 23:29:34.997524 531 raft_consensus.cc:482] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:34.997758 531 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:29:35.002497 531 raft_consensus.cc:504] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.004714 531 leader_election.cc:288] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 5 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:35.005254 904 raft_consensus.cc:979] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:35.005715 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:35.005959 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94efcd0874294a779f8c486e9e28ad75" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:35.006294 873 raft_consensus.cc:1067] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Signalling peer 7798fd3291924994932b048007322c25 to start an election
I20260927 23:29:35.006453 606 raft_consensus.cc:3032] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:29:35.006518 679 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:29:35.007002 606 raft_consensus.cc:729] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:35.008134 680 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2"
dest_uuid: "7798fd3291924994932b048007322c25"
from {username='slave'} at 127.0.0.1:48200
I20260927 23:29:35.007978 606 consensus_queue.cc:260] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.008719 680 raft_consensus.cc:482] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:35.009094 680 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.009464 606 raft_consensus.cc:3037] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:29:35.011780 679 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 5.
I20260927 23:29:35.012905 495 leader_election.cc:302] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:35.014623 913 raft_consensus.cc:2781] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:29:35.016048 606 raft_consensus.cc:2454] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 5.
I20260927 23:29:35.016088 680 raft_consensus.cc:504] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.017051 913 raft_consensus.cc:686] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 5 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:35.018009 913 consensus_queue.cc:237] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.019143 680 leader_election.cc:288] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Requested vote from peers 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343), 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691)
I20260927 23:29:35.019634 531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
I20260927 23:29:35.020234 531 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.020176 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fafcbb858a14e1198b5e3eb02e1c9d2" candidate_uuid: "7798fd3291924994932b048007322c25" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:35.020807 606 raft_consensus.cc:3032] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:35.021126 606 raft_consensus.cc:729] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:35.021857 606 consensus_queue.cc:260] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.022797 606 raft_consensus.cc:3037] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.026882 531 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:35.027014 425 catalog_manager.cc:5495] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 4 to 5, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 5 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:35.027959 645 leader_election.cc:302] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:35.028326 606 raft_consensus.cc:2454] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7798fd3291924994932b048007322c25 in term 3.
I20260927 23:29:35.028854 850 raft_consensus.cc:2781] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:35.029299 850 raft_consensus.cc:686] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Becoming Leader. State: Replica: 7798fd3291924994932b048007322c25, State: Running, Role: LEADER
I20260927 23:29:35.030076 850 consensus_queue.cc:237] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.037561 428 catalog_manager.cc:5495] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 reported cstate change: term changed from 2 to 3, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 7798fd3291924994932b048007322c25 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "7798fd3291924994932b048007322c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:35.100335 912 raft_consensus.cc:979] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:35.100793 912 raft_consensus.cc:1067] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:35.102394 531 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40888
I20260927 23:29:35.102833 531 raft_consensus.cc:482] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:35.103061 531 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:35.107618 531 raft_consensus.cc:504] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.109325 531 leader_election.cc:288] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 4 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:35.110035 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:35.110255 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbcdea459084f5796ecec2f8db827be" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:35.110617 606 raft_consensus.cc:3032] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:29:35.110913 680 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:35.110940 606 raft_consensus.cc:729] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:35.111608 606 consensus_queue.cc:260] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.112455 606 raft_consensus.cc:3037] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:29:35.115535 680 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 4.
I20260927 23:29:35.116509 606 raft_consensus.cc:2454] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 4.
I20260927 23:29:35.116488 495 leader_election.cc:302] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:35.117394 913 raft_consensus.cc:2781] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:29:35.118048 913 raft_consensus.cc:686] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 4 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:35.118918 913 consensus_queue.cc:237] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.126286 428 catalog_manager.cc:5495] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 3 to 4, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 4 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:35.248328 904 raft_consensus.cc:979] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: : Instructing follower 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:35.248759 912 raft_consensus.cc:1067] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Signalling peer 842a1c42cccc4b9390e0a4464a696512 to start an election
I20260927 23:29:35.250089 531 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1"
dest_uuid: "842a1c42cccc4b9390e0a4464a696512"
from {username='slave'} at 127.0.0.1:40888
I20260927 23:29:35.250576 531 raft_consensus.cc:482] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:35.250897 531 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.255326 531 raft_consensus.cc:504] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.256793 531 leader_election.cc:288] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 election: Requested vote from peers 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2:43691), 7798fd3291924994932b048007322c25 (127.0.99.3:36007)
I20260927 23:29:35.257879 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7798fd3291924994932b048007322c25"
I20260927 23:29:35.257925 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ead357ebc64eec911e793d35f180b1" candidate_uuid: "842a1c42cccc4b9390e0a4464a696512" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "794c8bb85c71461e9bb257c6afaec6eb"
I20260927 23:29:35.258469 680 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.258490 606 raft_consensus.cc:3032] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:35.258882 606 raft_consensus.cc:729] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 794c8bb85c71461e9bb257c6afaec6eb, State: Running, Role: LEADER
I20260927 23:29:35.259464 606 consensus_queue.cc:260] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.260270 606 raft_consensus.cc:3037] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:35.263128 680 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 3.
I20260927 23:29:35.264119 495 leader_election.cc:302] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7798fd3291924994932b048007322c25, 842a1c42cccc4b9390e0a4464a696512; no voters:
I20260927 23:29:35.264715 606 raft_consensus.cc:2454] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842a1c42cccc4b9390e0a4464a696512 in term 3.
I20260927 23:29:35.264816 913 raft_consensus.cc:2781] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:35.265285 913 raft_consensus.cc:686] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Becoming Leader. State: Replica: 842a1c42cccc4b9390e0a4464a696512, State: Running, Role: LEADER
I20260927 23:29:35.266041 913 consensus_queue.cc:237] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } }
I20260927 23:29:35.272532 428 catalog_manager.cc:5495] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 reported cstate change: term changed from 2 to 3, leader changed from 794c8bb85c71461e9bb257c6afaec6eb (127.0.99.2) to 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "842a1c42cccc4b9390e0a4464a696512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:35.413182 531 raft_consensus.cc:1261] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.414386 850 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:35.425166 606 raft_consensus.cc:1261] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.426363 850 consensus_queue.cc:1035] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:35.506105 679 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:29:35.507247 913 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:29:35.515950 606 raft_consensus.cc:1261] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:29:35.517280 925 consensus_queue.cc:1035] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:29:35.519191 606 raft_consensus.cc:1261] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.520427 850 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:35.530421 531 raft_consensus.cc:1261] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Refusing update from remote peer 7798fd3291924994932b048007322c25: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.531927 928 consensus_queue.cc:1035] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842a1c42cccc4b9390e0a4464a696512" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:29:35.533384 606 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 5 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:29:35.536418 914 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:29:35.554188 679 raft_consensus.cc:1261] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 5 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:29:35.556030 925 consensus_queue.cc:1035] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:29:35.739027 679 raft_consensus.cc:1261] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.740144 925 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7798fd3291924994932b048007322c25" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 36007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:35.748679 605 raft_consensus.cc:1261] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Refusing update from remote peer 842a1c42cccc4b9390e0a4464a696512: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:35.750393 934 consensus_queue.cc:1035] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "794c8bb85c71461e9bb257c6afaec6eb" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 43691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:29:35.818660 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:35.824862 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:35.825197 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:36.825958 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:36.831360 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:36.831717 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:37.832587 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:37.837385 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:37.837749 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:38.838778 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:38.844393 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:38.844779 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:39.845546 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:39.850688 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:39.851121 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:40.851923 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:40.856798 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:40.857204 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:41.858152 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:41.863878 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:41.864279 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:42.865026 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:42.869477 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:42.869797 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:43.870564 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:43.875200 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:43.875635 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:44.902105 396 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260927 23:29:44.902909 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20260927 23:29:44.930550 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:29:44.931263 396 tablet_replica.cc:330] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.931988 396 raft_consensus.cc:2229] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.932569 396 raft_consensus.cc:2258] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 842a1c42cccc4b9390e0a4464a696512 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.934906 396 tablet_replica.cc:330] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.935415 396 raft_consensus.cc:2229] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 5 LEADER]: Raft consensus shutting down.
I20260927 23:29:44.936208 396 raft_consensus.cc:2258] T 94efcd0874294a779f8c486e9e28ad75 P 842a1c42cccc4b9390e0a4464a696512 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.937872 396 tablet_replica.cc:330] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.938452 396 raft_consensus.cc:2229] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.938906 396 raft_consensus.cc:2258] T 26ae0bb2648b49cc96a48f44c5c446be P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.940490 396 tablet_replica.cc:330] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.940936 396 raft_consensus.cc:2229] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 4 LEADER]: Raft consensus shutting down.
I20260927 23:29:44.941709 396 raft_consensus.cc:2258] T 2bbcdea459084f5796ecec2f8db827be P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.943334 396 tablet_replica.cc:330] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.943801 396 raft_consensus.cc:2229] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.944213 396 raft_consensus.cc:2258] T 83126b384e0d437994105143a69490c9 P 842a1c42cccc4b9390e0a4464a696512 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.945808 396 tablet_replica.cc:330] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.946272 396 raft_consensus.cc:2229] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:29:44.947220 396 raft_consensus.cc:2258] T 34ead357ebc64eec911e793d35f180b1 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260927 23:29:44.948539 571 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260927 23:29:44.949353 396 tablet_replica.cc:330] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.949892 396 raft_consensus.cc:2229] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.950405 396 raft_consensus.cc:2258] T 8beb9a657b844a0ab14c93f43a140b5f P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260927 23:29:44.951817 571 consensus_peers.cc:487] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb -> Peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Couldn't send request to peer 842a1c42cccc4b9390e0a4464a696512. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:29:44.952436 396 tablet_replica.cc:330] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.952909 396 raft_consensus.cc:2229] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.953320 396 raft_consensus.cc:2258] T ba86a414c8484f1282985adfd9d4f0dd P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:44.954964 396 tablet_replica.cc:330] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512: stopping tablet replica
I20260927 23:29:44.955647 396 raft_consensus.cc:2229] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:44.956095 396 raft_consensus.cc:2258] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 842a1c42cccc4b9390e0a4464a696512 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.046905 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
W20260927 23:29:45.048049 645 consensus_peers.cc:487] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 -> Peer 842a1c42cccc4b9390e0a4464a696512 (127.0.99.1:44343): Couldn't send request to peer 842a1c42cccc4b9390e0a4464a696512. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.1:44343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 23:29:45.064934 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20260927 23:29:45.090941 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:29:45.091598 396 tablet_replica.cc:330] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.092170 396 raft_consensus.cc:2229] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.092981 396 raft_consensus.cc:2258] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 794c8bb85c71461e9bb257c6afaec6eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.094794 396 tablet_replica.cc:330] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.095276 396 raft_consensus.cc:2229] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.096050 396 raft_consensus.cc:2258] T 8beb9a657b844a0ab14c93f43a140b5f P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.097774 396 tablet_replica.cc:330] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.098233 396 raft_consensus.cc:2229] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.098733 396 raft_consensus.cc:2258] T 83126b384e0d437994105143a69490c9 P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.100227 396 tablet_replica.cc:330] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.100696 396 raft_consensus.cc:2229] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.101121 396 raft_consensus.cc:2258] T 34ead357ebc64eec911e793d35f180b1 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.102811 396 tablet_replica.cc:330] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.103327 396 raft_consensus.cc:2229] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.103732 396 raft_consensus.cc:2258] T 2bbcdea459084f5796ecec2f8db827be P 794c8bb85c71461e9bb257c6afaec6eb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.105185 396 tablet_replica.cc:330] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.105634 396 raft_consensus.cc:2229] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.106021 396 raft_consensus.cc:2258] T 94efcd0874294a779f8c486e9e28ad75 P 794c8bb85c71461e9bb257c6afaec6eb [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.107437 396 tablet_replica.cc:330] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.107856 396 raft_consensus.cc:2229] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.108280 396 raft_consensus.cc:2258] T 26ae0bb2648b49cc96a48f44c5c446be P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.109767 396 tablet_replica.cc:330] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.110189 396 raft_consensus.cc:2229] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.110626 396 raft_consensus.cc:2258] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.112381 396 tablet_replica.cc:330] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb: stopping tablet replica
I20260927 23:29:45.112797 396 raft_consensus.cc:2229] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.113785 396 raft_consensus.cc:2258] T ba86a414c8484f1282985adfd9d4f0dd P 794c8bb85c71461e9bb257c6afaec6eb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.144415 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20260927 23:29:45.160946 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20260927 23:29:45.192217 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:29:45.193015 396 tablet_replica.cc:330] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.193797 396 raft_consensus.cc:2229] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.194592 396 raft_consensus.cc:2258] T 26ae0bb2648b49cc96a48f44c5c446be P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.196139 396 tablet_replica.cc:330] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.196565 396 raft_consensus.cc:2229] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.196942 396 raft_consensus.cc:2258] T 2bbcdea459084f5796ecec2f8db827be P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.198415 396 tablet_replica.cc:330] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.198839 396 raft_consensus.cc:2229] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.199204 396 raft_consensus.cc:2258] T 94efcd0874294a779f8c486e9e28ad75 P 7798fd3291924994932b048007322c25 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.200713 396 tablet_replica.cc:330] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.201148 396 raft_consensus.cc:2229] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.201587 396 raft_consensus.cc:2258] T ba86a414c8484f1282985adfd9d4f0dd P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.203147 396 tablet_replica.cc:330] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.203547 396 raft_consensus.cc:2229] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.204223 396 raft_consensus.cc:2258] T 0fafcbb858a14e1198b5e3eb02e1c9d2 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.205668 396 tablet_replica.cc:330] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.206077 396 raft_consensus.cc:2229] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.206511 396 raft_consensus.cc:2258] T 34ead357ebc64eec911e793d35f180b1 P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.207882 396 tablet_replica.cc:330] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.208276 396 raft_consensus.cc:2229] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.208626 396 raft_consensus.cc:2258] T 8beb9a657b844a0ab14c93f43a140b5f P 7798fd3291924994932b048007322c25 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.210242 396 tablet_replica.cc:330] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.210750 396 raft_consensus.cc:2229] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 4 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.211428 396 raft_consensus.cc:2258] T 83126b384e0d437994105143a69490c9 P 7798fd3291924994932b048007322c25 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.212844 396 tablet_replica.cc:330] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25: stopping tablet replica
I20260927 23:29:45.213201 396 raft_consensus.cc:2229] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:29:45.213507 396 raft_consensus.cc:2258] T a4dd5edb0ddf4a649ebd29ca2df70e23 P 7798fd3291924994932b048007322c25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.240952 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20260927 23:29:45.258028 396 master.cc:537] Master@127.0.99.62:46049 shutting down...
I20260927 23:29:45.454083 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:29:45.454769 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:29:45.455670 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P aac04c8bfc4a4e88a80128e74708ff59: stopping tablet replica
I20260927 23:29:45.474037 396 master.cc:552] Master@127.0.99.62:46049 shutdown complete.
7798fd3291924994932b048007322c25 0
794c8bb85c71461e9bb257c6afaec6eb 4
842a1c42cccc4b9390e0a4464a696512 5
7798fd3291924994932b048007322c25 3
794c8bb85c71461e9bb257c6afaec6eb 3
842a1c42cccc4b9390e0a4464a696512 3
7798fd3291924994932b048007322c25 1
794c8bb85c71461e9bb257c6afaec6eb 8
842a1c42cccc4b9390e0a4464a696512 0
7798fd3291924994932b048007322c25 3
794c8bb85c71461e9bb257c6afaec6eb 3
842a1c42cccc4b9390e0a4464a696512 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31327 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260927 23:29:45.512042 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:37253
I20260927 23:29:45.513029 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:45.518090 944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.519785 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.519909 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:45.520375 396 server_base.cc:1034] running on GCE node
I20260927 23:29:45.521307 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:45.521502 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:45.521641 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551785521630 us; error 0 us; skew 500 ppm
I20260927 23:29:45.522071 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:45.524448 396 webserver.cc:460] Webserver started at http://127.0.99.62:46445/ using document root <none> and password file <none>
I20260927 23:29:45.524886 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:45.525035 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:45.525252 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:45.526471 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/master-0-root/instance:
uuid: "1d3c5d2f284d484bbe6a841cc2a782e2"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.530773 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:29:45.534011 952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.534899 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:29:45.535310 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/master-0-root
uuid: "1d3c5d2f284d484bbe6a841cc2a782e2"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.535635 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:45.555693 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:45.557036 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:45.595997 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:37253
I20260927 23:29:45.596062 1003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:37253 every 8 connection(s)
I20260927 23:29:45.600358 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:45.609285 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2: Bootstrap starting.
I20260927 23:29:45.613358 1004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:45.617367 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2: No bootstrap required, opened a new log
I20260927 23:29:45.619259 1004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER }
I20260927 23:29:45.619709 1004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:45.619880 1004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3c5d2f284d484bbe6a841cc2a782e2, State: Initialized, Role: FOLLOWER
I20260927 23:29:45.620416 1004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER }
I20260927 23:29:45.620826 1004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:29:45.621009 1004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:29:45.621218 1004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:45.624821 1004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER }
I20260927 23:29:45.625257 1004 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3c5d2f284d484bbe6a841cc2a782e2; no voters:
I20260927 23:29:45.626303 1004 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:29:45.626771 1007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:45.628041 1007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3c5d2f284d484bbe6a841cc2a782e2, State: Running, Role: LEADER
I20260927 23:29:45.628708 1007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER }
I20260927 23:29:45.629190 1004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:29:45.633219 1008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER } }
I20260927 23:29:45.633984 1008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [sys.catalog]: This master's current role is: LEADER
I20260927 23:29:45.634076 1009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3c5d2f284d484bbe6a841cc2a782e2. Latest consensus state: current_term: 1 leader_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3c5d2f284d484bbe6a841cc2a782e2" member_type: VOTER } }
I20260927 23:29:45.635069 1009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [sys.catalog]: This master's current role is: LEADER
I20260927 23:29:45.639945 1014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:29:45.644465 1014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:29:45.647279 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 23:29:45.652817 1014 catalog_manager.cc:1347] Generated new cluster ID: 34a50fdb675c4a0b9780a3485758d7be
I20260927 23:29:45.653090 1014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:29:45.674182 1014 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:29:45.675668 1014 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:29:45.685655 1014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2: Generated new TSK 0
I20260927 23:29:45.686244 1014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:29:45.713821 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:45.720423 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.721047 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.722224 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:45.722254 396 server_base.cc:1034] running on GCE node
I20260927 23:29:45.723521 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:45.723724 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:45.723892 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551785723878 us; error 0 us; skew 500 ppm
I20260927 23:29:45.724400 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:45.726925 396 webserver.cc:460] Webserver started at http://127.0.99.1:33117/ using document root <none> and password file <none>
I20260927 23:29:45.727356 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:45.727504 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:45.727699 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:45.728721 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-0-root/instance:
uuid: "1e501af30d1b42598487d926ba2719fd"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.733486 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 23:29:45.736783 1033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.737573 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:29:45.737913 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-0-root
uuid: "1e501af30d1b42598487d926ba2719fd"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.738214 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:45.750818 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:45.751900 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:45.753355 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:45.755549 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:45.755720 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.755995 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:45.756170 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.795238 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:32985
I20260927 23:29:45.795338 1095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:32985 every 8 connection(s)
I20260927 23:29:45.799650 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:45.807054 1100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.808879 1101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.812160 1103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:45.814565 396 server_base.cc:1034] running on GCE node
I20260927 23:29:45.815557 1096 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37253
I20260927 23:29:45.815939 1096 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:45.816076 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:45.816325 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:45.816475 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551785816466 us; error 0 us; skew 500 ppm
I20260927 23:29:45.816774 1096 heartbeater.cc:511] Master 127.0.99.62:37253 requested a full tablet report, sending...
I20260927 23:29:45.817080 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:45.819021 969 ts_manager.cc:194] Registered new tserver with Master: 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985)
I20260927 23:29:45.819914 396 webserver.cc:460] Webserver started at http://127.0.99.2:38707/ using document root <none> and password file <none>
I20260927 23:29:45.820562 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:45.820797 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:45.821118 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:45.821859 969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34916
I20260927 23:29:45.822615 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-1-root/instance:
uuid: "789a6d94ecf34e1daf6aa60f4175672b"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.827191 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 23:29:45.830443 1108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.831337 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:29:45.831622 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-1-root
uuid: "789a6d94ecf34e1daf6aa60f4175672b"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.831905 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:45.843981 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:45.845196 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:45.846668 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:45.848745 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:45.848954 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.849179 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:45.849332 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.887984 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:42071
I20260927 23:29:45.888077 1170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:42071 every 8 connection(s)
I20260927 23:29:45.892508 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:29:45.899441 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:29:45.900545 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:45.903437 396 server_base.cc:1034] running on GCE node
W20260927 23:29:45.904213 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:29:45.905284 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:29:45.905540 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:29:45.905745 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551785905726 us; error 0 us; skew 500 ppm
I20260927 23:29:45.906463 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:29:45.906844 1171 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37253
I20260927 23:29:45.907191 1171 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:45.908079 1171 heartbeater.cc:511] Master 127.0.99.62:37253 requested a full tablet report, sending...
I20260927 23:29:45.909086 396 webserver.cc:460] Webserver started at http://127.0.99.3:39257/ using document root <none> and password file <none>
I20260927 23:29:45.909636 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:29:45.909833 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:29:45.910105 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:29:45.910259 969 ts_manager.cc:194] Registered new tserver with Master: 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:45.911307 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-2-root/instance:
uuid: "35739abf9980475798a7457f7dfac761"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.911824 969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34932
I20260927 23:29:45.915849 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:29:45.919217 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.920099 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 23:29:45.920406 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-2-root
uuid: "35739abf9980475798a7457f7dfac761"
format_stamp: "Formatted at 2026-09-27 23:29:45 on dist-test-slave-zkzt"
I20260927 23:29:45.920712 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790551754024304-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:29:45.934859 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:29:45.936111 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:29:45.937600 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:29:45.939610 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:29:45.939813 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.940052 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:29:45.940213 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:29:45.979025 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:33021
I20260927 23:29:45.979161 1244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:33021 every 8 connection(s)
I20260927 23:29:45.993299 1245 heartbeater.cc:347] Connected to a master server at 127.0.99.62:37253
I20260927 23:29:45.993666 1245 heartbeater.cc:464] Registering TS with master...
I20260927 23:29:45.994354 1245 heartbeater.cc:511] Master 127.0.99.62:37253 requested a full tablet report, sending...
I20260927 23:29:45.996104 969 ts_manager.cc:194] Registered new tserver with Master: 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:45.996263 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014352861s
I20260927 23:29:45.997684 969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34936
I20260927 23:29:45.998764 396 test_util.cc:268] Using random seed: 1395132824
I20260927 23:29:46.021870 969 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34948:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:29:46.025111 969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:29:46.113839 1209 tablet_service.cc:1467] Processing CreateTablet for tablet 31555b5cc778446e93141a07ead471e5 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 23:29:46.115375 1209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31555b5cc778446e93141a07ead471e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.115976 1203 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a9dc40e674fd6a5430bfd0aa9f911 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 23:29:46.116389 1202 tablet_service.cc:1467] Processing CreateTablet for tablet 892b90646ced48509346c4d155d805e8 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 23:29:46.117348 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9a9dc40e674fd6a5430bfd0aa9f911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.118011 1202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892b90646ced48509346c4d155d805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.117791 1208 tablet_service.cc:1467] Processing CreateTablet for tablet 654b86b5107e4105a9ab44ecd0e3f13a (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 23:29:46.119065 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b86b5107e4105a9ab44ecd0e3f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.122273 1207 tablet_service.cc:1467] Processing CreateTablet for tablet b4ed00c8fdda44fdba9d7e79f076f6d6 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 23:29:46.123628 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ed00c8fdda44fdba9d7e79f076f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.127522 1206 tablet_service.cc:1467] Processing CreateTablet for tablet 59d5cee9e4fc45c790038239db69baf4 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 23:29:46.128849 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d5cee9e4fc45c790038239db69baf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.131428 1201 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4b4aa3a364d96805f63e9706818f7 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 23:29:46.132663 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4b4aa3a364d96805f63e9706818f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.135914 1204 tablet_service.cc:1467] Processing CreateTablet for tablet fa1cfd87a16a4bdd97a15922ebd50465 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 23:29:46.137640 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1cfd87a16a4bdd97a15922ebd50465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.139595 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 0949c6f71ad54c289d0a40b1a057bedd (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 23:29:46.142125 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0949c6f71ad54c289d0a40b1a057bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.144599 1260 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.113969 1210 tablet_service.cc:1467] Processing CreateTablet for tablet 8313caee40fa4473a5c34cbf0f6e0e2b (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 23:29:46.147140 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313caee40fa4473a5c34cbf0f6e0e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.151374 1136 tablet_service.cc:1467] Processing CreateTablet for tablet 8313caee40fa4473a5c34cbf0f6e0e2b (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 23:29:46.152704 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313caee40fa4473a5c34cbf0f6e0e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.152446 1134 tablet_service.cc:1467] Processing CreateTablet for tablet 654b86b5107e4105a9ab44ecd0e3f13a (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 23:29:46.154743 1135 tablet_service.cc:1467] Processing CreateTablet for tablet 31555b5cc778446e93141a07ead471e5 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 23:29:46.155503 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 59d5cee9e4fc45c790038239db69baf4 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 23:29:46.157009 1127 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4b4aa3a364d96805f63e9706818f7 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 23:29:46.157598 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 0949c6f71ad54c289d0a40b1a057bedd (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 23:29:46.159539 1130 tablet_service.cc:1467] Processing CreateTablet for tablet fa1cfd87a16a4bdd97a15922ebd50465 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 23:29:46.161496 1129 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a9dc40e674fd6a5430bfd0aa9f911 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 23:29:46.163429 1128 tablet_service.cc:1467] Processing CreateTablet for tablet 892b90646ced48509346c4d155d805e8 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 23:29:46.165514 1061 tablet_service.cc:1467] Processing CreateTablet for tablet 8313caee40fa4473a5c34cbf0f6e0e2b (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 23:29:46.166996 1061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8313caee40fa4473a5c34cbf0f6e0e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.167541 1053 tablet_service.cc:1467] Processing CreateTablet for tablet 892b90646ced48509346c4d155d805e8 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 23:29:46.168823 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892b90646ced48509346c4d155d805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.171494 1060 tablet_service.cc:1467] Processing CreateTablet for tablet 31555b5cc778446e93141a07ead471e5 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 23:29:46.172214 1054 tablet_service.cc:1467] Processing CreateTablet for tablet cc9a9dc40e674fd6a5430bfd0aa9f911 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 23:29:46.172897 1060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31555b5cc778446e93141a07ead471e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.173456 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9a9dc40e674fd6a5430bfd0aa9f911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.171489 1052 tablet_service.cc:1467] Processing CreateTablet for tablet bfd4b4aa3a364d96805f63e9706818f7 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 23:29:46.175639 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4b4aa3a364d96805f63e9706818f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.175506 1059 tablet_service.cc:1467] Processing CreateTablet for tablet 654b86b5107e4105a9ab44ecd0e3f13a (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 23:29:46.176831 1059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b86b5107e4105a9ab44ecd0e3f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.179538 1058 tablet_service.cc:1467] Processing CreateTablet for tablet b4ed00c8fdda44fdba9d7e79f076f6d6 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 23:29:46.180750 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ed00c8fdda44fdba9d7e79f076f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.181495 1260 tablet_bootstrap.cc:654] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.183495 1057 tablet_service.cc:1467] Processing CreateTablet for tablet 59d5cee9e4fc45c790038239db69baf4 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 23:29:46.184732 1057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d5cee9e4fc45c790038239db69baf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.189677 1056 tablet_service.cc:1467] Processing CreateTablet for tablet 0949c6f71ad54c289d0a40b1a057bedd (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 23:29:46.193414 1055 tablet_service.cc:1467] Processing CreateTablet for tablet fa1cfd87a16a4bdd97a15922ebd50465 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 23:29:46.197469 1133 tablet_service.cc:1467] Processing CreateTablet for tablet b4ed00c8fdda44fdba9d7e79f076f6d6 (DEFAULT_TABLE table=test-workload [id=6a1e0824ce6d4168b27f786714c96767]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 23:29:46.198875 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4ed00c8fdda44fdba9d7e79f076f6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.199798 1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31555b5cc778446e93141a07ead471e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.201771 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9a9dc40e674fd6a5430bfd0aa9f911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.203922 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1cfd87a16a4bdd97a15922ebd50465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.207463 1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654b86b5107e4105a9ab44ecd0e3f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.208281 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1cfd87a16a4bdd97a15922ebd50465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.209815 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0949c6f71ad54c289d0a40b1a057bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.209856 1128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892b90646ced48509346c4d155d805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.212973 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd4b4aa3a364d96805f63e9706818f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.213729 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d5cee9e4fc45c790038239db69baf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.215127 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0949c6f71ad54c289d0a40b1a057bedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:29:46.253582 1260 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.254158 1260 ts_tablet_manager.cc:1397] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.110s user 0.024s sys 0.029s
I20260927 23:29:46.256810 1260 raft_consensus.cc:348] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.257690 1260 raft_consensus.cc:374] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.258028 1260 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.258899 1260 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.329429 1260 ts_tablet_manager.cc:1428] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.075s user 0.019s sys 0.027s
I20260927 23:29:46.330400 1245 heartbeater.cc:503] Master 127.0.99.62:37253 was elected leader, sending a full tablet report...
I20260927 23:29:46.332141 1260 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.329820 1263 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.333333 1264 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.339610 1260 tablet_bootstrap.cc:654] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.339922 1263 tablet_bootstrap.cc:654] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.343205 1264 tablet_bootstrap.cc:654] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.352273 1263 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.352273 1260 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.353165 1263 ts_tablet_manager.cc:1397] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.000s
I20260927 23:29:46.353206 1260 ts_tablet_manager.cc:1397] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20260927 23:29:46.354478 1264 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.354951 1264 ts_tablet_manager.cc:1397] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.014s
I20260927 23:29:46.356174 1260 raft_consensus.cc:348] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.356302 1263 raft_consensus.cc:348] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.356942 1260 raft_consensus.cc:374] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.357246 1263 raft_consensus.cc:374] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.357263 1260 raft_consensus.cc:729] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.357482 1264 raft_consensus.cc:348] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.357913 1263 raft_consensus.cc:729] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.358355 1264 raft_consensus.cc:374] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.358741 1264 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.358393 1260 consensus_queue.cc:260] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.359135 1263 consensus_queue.cc:260] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.360857 1260 ts_tablet_manager.cc:1428] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:29:46.361716 1260 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.366761 1263 ts_tablet_manager.cc:1428] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.013s user 0.009s sys 0.001s
I20260927 23:29:46.367688 1260 tablet_bootstrap.cc:654] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.367781 1096 heartbeater.cc:503] Master 127.0.99.62:37253 was elected leader, sending a full tablet report...
I20260927 23:29:46.372226 1260 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.372655 1260 ts_tablet_manager.cc:1397] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 23:29:46.373569 1263 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.375556 1260 raft_consensus.cc:348] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.376510 1260 raft_consensus.cc:374] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.376825 1260 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.377686 1264 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.377609 1260 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.378851 1268 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.379462 1268 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.380487 1260 ts_tablet_manager.cc:1428] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 23:29:46.386917 1171 heartbeater.cc:503] Master 127.0.99.62:37253 was elected leader, sending a full tablet report...
I20260927 23:29:46.387202 1264 ts_tablet_manager.cc:1428] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.032s user 0.021s sys 0.011s
I20260927 23:29:46.387589 1260 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.388135 1264 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.389454 1268 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.393821 1264 tablet_bootstrap.cc:654] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.394035 1260 tablet_bootstrap.cc:654] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.385598 1263 tablet_bootstrap.cc:654] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.401496 1264 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.402258 1264 ts_tablet_manager.cc:1397] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260927 23:29:46.405958 1264 raft_consensus.cc:348] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.407159 1264 raft_consensus.cc:374] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.407595 1264 raft_consensus.cc:729] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.408596 1264 consensus_queue.cc:260] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.412122 1264 ts_tablet_manager.cc:1428] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260927 23:29:46.413401 1264 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.421588 1264 tablet_bootstrap.cc:654] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.425048 1268 raft_consensus.cc:482] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.425700 1268 raft_consensus.cc:504] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.428265 1268 leader_election.cc:288] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.434378 1264 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.435102 1264 ts_tablet_manager.cc:1397] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20260927 23:29:46.439122 1264 raft_consensus.cc:348] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.439634 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.439669 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
W20260927 23:29:46.443077 1112 leader_election.cc:341] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:29:46.444051 1112 leader_election.cc:341] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.445158 1264 raft_consensus.cc:374] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.445696 1264 raft_consensus.cc:729] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.446837 1264 consensus_queue.cc:260] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.450243 1264 ts_tablet_manager.cc:1428] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260927 23:29:46.451576 1264 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.459903 1264 tablet_bootstrap.cc:654] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.473119 1264 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.473884 1264 ts_tablet_manager.cc:1397] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.005s
I20260927 23:29:46.477366 1264 raft_consensus.cc:348] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.478658 1264 raft_consensus.cc:374] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.479104 1264 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.480190 1264 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.483434 1264 ts_tablet_manager.cc:1428] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260927 23:29:46.484742 1264 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.485965 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.487636 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
W20260927 23:29:46.489351 1110 leader_election.cc:341] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.489809 1110 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761
W20260927 23:29:46.490981 1110 leader_election.cc:341] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.491417 1110 leader_election.cc:302] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761
I20260927 23:29:46.492480 1268 raft_consensus.cc:2726] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:46.492952 1271 raft_consensus.cc:2726] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:46.493522 1264 tablet_bootstrap.cc:654] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.501173 1263 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.501700 1263 ts_tablet_manager.cc:1397] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.128s user 0.054s sys 0.050s
I20260927 23:29:46.503202 1260 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.503774 1260 ts_tablet_manager.cc:1397] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.116s user 0.023s sys 0.051s
I20260927 23:29:46.505283 1264 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.505729 1264 ts_tablet_manager.cc:1397] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20260927 23:29:46.506095 1263 raft_consensus.cc:348] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.506943 1263 raft_consensus.cc:374] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.507313 1263 raft_consensus.cc:729] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.506774 1260 raft_consensus.cc:348] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.507855 1260 raft_consensus.cc:374] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.508244 1260 raft_consensus.cc:729] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.508090 1263 consensus_queue.cc:260] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.508324 1264 raft_consensus.cc:348] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.509065 1264 raft_consensus.cc:374] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.509339 1264 raft_consensus.cc:729] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.510512 1263 ts_tablet_manager.cc:1428] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260927 23:29:46.511487 1263 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.510007 1264 consensus_queue.cc:260] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.524416 1264 ts_tablet_manager.cc:1428] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260927 23:29:46.524341 1260 consensus_queue.cc:260] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.525456 1264 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.527235 1263 tablet_bootstrap.cc:654] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.530313 1260 ts_tablet_manager.cc:1428] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260927 23:29:46.531284 1260 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.531606 1264 tablet_bootstrap.cc:654] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.538865 1260 tablet_bootstrap.cc:654] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.546100 1264 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.546849 1264 ts_tablet_manager.cc:1397] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20260927 23:29:46.548043 1263 tablet_bootstrap.cc:492] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.548535 1263 ts_tablet_manager.cc:1397] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.005s
I20260927 23:29:46.550047 1264 raft_consensus.cc:348] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.551043 1264 raft_consensus.cc:374] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.551430 1264 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.551306 1263 raft_consensus.cc:348] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.552157 1263 raft_consensus.cc:374] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.552466 1263 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.552567 1264 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.553160 1263 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.555505 1263 ts_tablet_manager.cc:1428] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 23:29:46.555538 1264 ts_tablet_manager.cc:1428] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 23:29:46.556517 1263 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.556831 1264 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.580799 1263 tablet_bootstrap.cc:654] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.587888 1264 tablet_bootstrap.cc:654] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.596164 1263 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.596668 1263 ts_tablet_manager.cc:1397] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.004s
I20260927 23:29:46.599395 1263 raft_consensus.cc:348] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.600221 1263 raft_consensus.cc:374] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.600507 1263 raft_consensus.cc:729] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.601236 1263 consensus_queue.cc:260] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.605171 1263 ts_tablet_manager.cc:1428] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260927 23:29:46.605610 1264 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.606283 1263 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.606348 1264 ts_tablet_manager.cc:1397] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.016s
I20260927 23:29:46.610122 1264 raft_consensus.cc:348] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.611199 1264 raft_consensus.cc:374] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.612102 1264 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.613188 1263 tablet_bootstrap.cc:654] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.612999 1264 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.686060 1262 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.686628 1262 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.688712 1262 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.696102 1262 raft_consensus.cc:482] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.696609 1262 raft_consensus.cc:504] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.698942 1262 leader_election.cc:288] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.617895 1264 ts_tablet_manager.cc:1428] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260927 23:29:46.700785 1264 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.705579 1260 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.706089 1260 ts_tablet_manager.cc:1397] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.175s user 0.029s sys 0.037s
I20260927 23:29:46.706712 1264 tablet_bootstrap.cc:654] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.709136 1260 raft_consensus.cc:348] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.709923 1260 raft_consensus.cc:374] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.710232 1260 raft_consensus.cc:729] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.710994 1260 consensus_queue.cc:260] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.711997 1264 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.712484 1264 ts_tablet_manager.cc:1397] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 23:29:46.713960 1260 ts_tablet_manager.cc:1428] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260927 23:29:46.715075 1260 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.714984 1264 raft_consensus.cc:348] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.715732 1264 raft_consensus.cc:374] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.716004 1264 raft_consensus.cc:729] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.716723 1264 consensus_queue.cc:260] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.718765 1264 ts_tablet_manager.cc:1428] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260927 23:29:46.719587 1264 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.724805 1264 tablet_bootstrap.cc:654] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
W20260927 23:29:46.725834 1017 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:29:46.731704 1260 tablet_bootstrap.cc:654] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.732702 1263 tablet_bootstrap.cc:492] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.733219 1263 ts_tablet_manager.cc:1397] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.009s
I20260927 23:29:46.738376 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.738771 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.739395 1071 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
W20260927 23:29:46.740556 1186 leader_election.cc:341] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.741243 1186 leader_election.cc:302] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.741978 1262 raft_consensus.cc:2781] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.742393 1262 raft_consensus.cc:482] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.742762 1262 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.744249 1263 raft_consensus.cc:348] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.744793 1263 raft_consensus.cc:374] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.745035 1263 raft_consensus.cc:729] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.745608 1263 consensus_queue.cc:260] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.747478 1263 ts_tablet_manager.cc:1428] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260927 23:29:46.748193 1263 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.749969 1262 raft_consensus.cc:504] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.752327 1260 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.752816 1260 ts_tablet_manager.cc:1397] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.003s
I20260927 23:29:46.752938 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:46.753734 1071 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.755460 1260 raft_consensus.cc:348] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.756186 1260 raft_consensus.cc:374] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.756464 1260 raft_consensus.cc:729] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.757074 1262 leader_election.cc:288] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.757745 1260 consensus_queue.cc:260] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.760318 1260 ts_tablet_manager.cc:1428] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:29:46.761366 1260 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.762604 1071 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.768491 1262 raft_consensus.cc:482] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.768993 1262 raft_consensus.cc:504] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.769322 1269 raft_consensus.cc:482] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.769873 1269 raft_consensus.cc:504] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.770517 1186 leader_election.cc:302] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.770970 1262 leader_election.cc:288] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.771442 1262 raft_consensus.cc:2781] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.771988 1262 raft_consensus.cc:686] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:46.772228 1269 leader_election.cc:288] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.773080 1262 consensus_queue.cc:237] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.760354 1263 tablet_bootstrap.cc:654] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.780429 1264 tablet_bootstrap.cc:492] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:46.782015 1264 ts_tablet_manager.cc:1397] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.009s
I20260927 23:29:46.782704 1260 tablet_bootstrap.cc:654] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.789263 1263 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.789911 1263 ts_tablet_manager.cc:1397] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20260927 23:29:46.786064 1264 raft_consensus.cc:348] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.793797 1264 raft_consensus.cc:374] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.796077 1264 raft_consensus.cc:729] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.797538 1264 consensus_queue.cc:260] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.798074 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4b4aa3a364d96805f63e9706818f7" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.796391 1260 tablet_bootstrap.cc:492] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.799773 1071 raft_consensus.cc:2454] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.800010 1260 ts_tablet_manager.cc:1397] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.001s
I20260927 23:29:46.800638 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.801009 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.801295 1146 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.801591 1145 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.802877 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.802775 1260 raft_consensus.cc:348] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.803493 1145 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.803572 1260 raft_consensus.cc:374] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.803958 1260 raft_consensus.cc:729] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.804780 1271 raft_consensus.cc:482] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.804641 1260 consensus_queue.cc:260] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.805231 1271 raft_consensus.cc:504] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.806968 1260 ts_tablet_manager.cc:1428] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260927 23:29:46.807240 1271 leader_election.cc:288] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.807863 1260 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.807919 967 catalog_manager.cc:5495] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:46.808861 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.809656 1219 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 0.
I20260927 23:29:46.793183 1263 raft_consensus.cc:348] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.810396 1145 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.810488 1263 raft_consensus.cc:374] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.810901 1263 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.810983 1110 leader_election.cc:302] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:46.811921 1271 raft_consensus.cc:2781] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.812469 1271 raft_consensus.cc:482] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.812860 1271 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.811902 1186 leader_election.cc:302] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.811666 1263 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.814955 1184 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:46.816749 1263 ts_tablet_manager.cc:1428] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.027s user 0.008s sys 0.001s
I20260927 23:29:46.816193 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.817652 1263 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.814296 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4b4aa3a364d96805f63e9706818f7" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.818540 1145 raft_consensus.cc:2454] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
W20260927 23:29:46.820537 1112 leader_election.cc:341] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.820751 1260 tablet_bootstrap.cc:654] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.824039 1268 raft_consensus.cc:482] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.824502 1268 raft_consensus.cc:504] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.826648 1268 leader_election.cc:288] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.828341 1262 raft_consensus.cc:2781] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.828464 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.828740 1262 raft_consensus.cc:482] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.829056 1262 raft_consensus.cc:3037] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:29:46.829761 1110 leader_election.cc:341] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.831020 1291 raft_consensus.cc:2781] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.831354 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.836055 1260 tablet_bootstrap.cc:492] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.835331 1291 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.836609 1260 ts_tablet_manager.cc:1397] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.007s
W20260927 23:29:46.837944 1112 leader_election.cc:341] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.838421 1112 leader_election.cc:302] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761
I20260927 23:29:46.839251 1260 raft_consensus.cc:348] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.839903 1260 raft_consensus.cc:374] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.840157 1260 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.840819 1264 ts_tablet_manager.cc:1428] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.058s user 0.006s sys 0.001s
I20260927 23:29:46.836582 1291 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.839583 1268 raft_consensus.cc:2726] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:46.845008 1264 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: Bootstrap starting.
I20260927 23:29:46.835018 1263 tablet_bootstrap.cc:654] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.840864 1260 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.840198 1262 raft_consensus.cc:504] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.850299 1264 tablet_bootstrap.cc:654] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.850435 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4b4aa3a364d96805f63e9706818f7" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:46.851290 1071 raft_consensus.cc:3037] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.853796 1271 raft_consensus.cc:504] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.856523 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
W20260927 23:29:46.857820 1112 leader_election.cc:341] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.859655 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:46.860199 1219 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.860988 1071 raft_consensus.cc:2454] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.862447 1186 leader_election.cc:302] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.864382 1291 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.867578 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.865849 1303 raft_consensus.cc:482] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.868165 1145 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 0.
I20260927 23:29:46.867384 1260 ts_tablet_manager.cc:1428] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260927 23:29:46.869216 1260 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.869392 1035 leader_election.cc:302] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:46.870215 1269 raft_consensus.cc:2781] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.870590 1269 raft_consensus.cc:482] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.870893 1269 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.875095 1260 tablet_bootstrap.cc:654] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.877874 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.876240 1262 leader_election.cc:288] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.868248 1303 raft_consensus.cc:504] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.878517 1146 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.880334 1303 leader_election.cc:288] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.881134 1303 raft_consensus.cc:2781] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.881577 1303 raft_consensus.cc:686] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:46.882433 1303 consensus_queue.cc:237] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.877204 1291 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.883229 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.885185 1144 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.885113 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.876797 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd4b4aa3a364d96805f63e9706818f7" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.886144 1145 raft_consensus.cc:3037] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.887178 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.887972 1220 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 0.
I20260927 23:29:46.888173 1184 leader_election.cc:302] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:46.890280 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:46.890648 1291 raft_consensus.cc:2781] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.891113 1291 raft_consensus.cc:482] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.891537 1291 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:29:46.893520 1186 leader_election.cc:341] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.896277 1306 raft_consensus.cc:482] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.896764 1306 raft_consensus.cc:504] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.897362 1269 raft_consensus.cc:504] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.898918 1070 raft_consensus.cc:2379] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35739abf9980475798a7457f7dfac761 in current term 1: Already voted for candidate 1e501af30d1b42598487d926ba2719fd in this term.
I20260927 23:29:46.901938 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.902585 1220 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 0.
I20260927 23:29:46.903118 1263 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.902987 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.903616 1263 ts_tablet_manager.cc:1397] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.086s user 0.024s sys 0.023s
I20260927 23:29:46.904201 1035 leader_election.cc:302] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.907742 1146 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.909021 1184 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:46.910274 1301 raft_consensus.cc:2781] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.910677 1219 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 1.
I20260927 23:29:46.910823 1301 raft_consensus.cc:686] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:46.912154 1110 leader_election.cc:302] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:46.913522 1263 raft_consensus.cc:348] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.914258 1263 raft_consensus.cc:374] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.914558 1263 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.915223 1263 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.916857 1268 raft_consensus.cc:2781] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.911999 1301 consensus_queue.cc:237] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.919008 1268 raft_consensus.cc:686] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:46.919740 1303 raft_consensus.cc:482] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.920008 1268 consensus_queue.cc:237] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.920257 1303 raft_consensus.cc:504] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.923377 1303 leader_election.cc:288] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.924443 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.925307 1070 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.926506 1310 raft_consensus.cc:2781] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.926633 1186 leader_election.cc:302] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.927584 1263 ts_tablet_manager.cc:1428] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.024s user 0.014s sys 0.001s
I20260927 23:29:46.928463 1303 raft_consensus.cc:2781] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.928864 1303 raft_consensus.cc:482] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.929255 1303 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.931732 1263 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:46.928124 1310 raft_consensus.cc:482] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.932842 1271 leader_election.cc:288] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.932770 1310 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.936575 1269 leader_election.cc:288] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.939314 1263 tablet_bootstrap.cc:654] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.936681 1306 leader_election.cc:288] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.938130 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:46.944265 1146 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:46.938917 1145 raft_consensus.cc:2454] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.948221 1303 raft_consensus.cc:504] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:46.959795 1260 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
I20260927 23:29:46.960258 1260 ts_tablet_manager.cc:1397] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.018s
I20260927 23:29:46.960419 1310 raft_consensus.cc:504] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.962567 1260 raft_consensus.cc:348] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.963747 1260 raft_consensus.cc:374] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:46.963971 1260 raft_consensus.cc:729] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.964488 1260 consensus_queue.cc:260] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.968494 1303 leader_election.cc:288] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.968534 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:46.969172 1070 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.969457 1268 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:46.969308 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.967108 1310 leader_election.cc:288] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
W20260927 23:29:46.966596 1035 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:33021, user_credentials={real_user=slave}} blocked reactor thread for 62124.1us
I20260927 23:29:46.969961 1268 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.972858 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.973402 1146 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.974010 1268 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:46.974737 1291 raft_consensus.cc:504] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.975544 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:46.976176 1219 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 0.
I20260927 23:29:46.970227 1145 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.977906 1110 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:46.979317 1260 ts_tablet_manager.cc:1428] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260927 23:29:46.979326 1268 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:46.980383 1268 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:46.980595 1260 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: Bootstrap starting.
I20260927 23:29:46.980726 1268 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.981317 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
W20260927 23:29:46.982954 1035 leader_election.cc:341] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:29:46.983863 1035 leader_election.cc:341] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:46.985749 1146 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 1.
I20260927 23:29:46.985875 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:46.986809 1071 raft_consensus.cc:2379] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 35739abf9980475798a7457f7dfac761 in current term 1: Already voted for candidate 1e501af30d1b42598487d926ba2719fd in this term.
I20260927 23:29:46.986756 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:46.987365 1220 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:46.985875 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:46.988725 1035 leader_election.cc:302] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:46.989936 1070 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:46.990485 1291 leader_election.cc:288] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:46.969172 967 catalog_manager.cc:5495] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 0 to 1, leader changed from <none> to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 1 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:46.990972 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:46.992079 1146 raft_consensus.cc:2379] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 35739abf9980475798a7457f7dfac761 in current term 1: Already voted for candidate 1e501af30d1b42598487d926ba2719fd in this term.
I20260927 23:29:46.991053 1310 raft_consensus.cc:2781] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.991080 1219 raft_consensus.cc:2379] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1e501af30d1b42598487d926ba2719fd in current term 1: Already voted for candidate 35739abf9980475798a7457f7dfac761 in this term.
I20260927 23:29:46.992208 1186 leader_election.cc:302] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:46.992645 1310 raft_consensus.cc:686] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:46.993341 1291 raft_consensus.cc:2781] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:46.993821 1291 raft_consensus.cc:686] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:46.993453 1310 consensus_queue.cc:237] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:46.994766 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:46.996754 1260 tablet_bootstrap.cc:654] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:46.996969 1263 tablet_bootstrap.cc:492] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:46.997411 1263 ts_tablet_manager.cc:1397] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.066s user 0.019s sys 0.002s
I20260927 23:29:46.995364 1071 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 0.
I20260927 23:29:47.000777 1184 leader_election.cc:302] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35739abf9980475798a7457f7dfac761; no voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.003564 1301 raft_consensus.cc:2726] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 23:29:47.004832 1263 raft_consensus.cc:348] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.005318 1263 raft_consensus.cc:374] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:47.005475 1263 raft_consensus.cc:729] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:46.994800 1291 consensus_queue.cc:237] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:47.016537 966 catalog_manager.cc:5495] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.020895 1264 tablet_bootstrap.cc:492] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: No bootstrap required, opened a new log
I20260927 23:29:47.021428 1264 ts_tablet_manager.cc:1397] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent bootstrapping tablet: real 0.177s user 0.015s sys 0.000s
I20260927 23:29:47.006958 1263 consensus_queue.cc:260] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.023742 1268 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.025873 1268 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.027043 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.027562 1220 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 1.
I20260927 23:29:47.027768 1071 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.029042 1310 raft_consensus.cc:482] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.029402 1263 ts_tablet_manager.cc:1428] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260927 23:29:47.028012 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.030995 1263 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:47.030897 1219 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.029474 1310 raft_consensus.cc:504] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.019632 965 catalog_manager.cc:5495] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:47.034559 1310 leader_election.cc:288] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.035238 1035 leader_election.cc:302] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters: 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.036520 1310 raft_consensus.cc:2781] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:47.036854 1291 raft_consensus.cc:482] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.038072 1291 raft_consensus.cc:504] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.039388 1219 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 1.
I20260927 23:29:47.037189 1145 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:47.040483 1110 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.039952 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:47.035609 965 catalog_manager.cc:5495] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:47.042809 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:47.040591 1260 tablet_bootstrap.cc:492] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: No bootstrap required, opened a new log
W20260927 23:29:47.044099 1186 leader_election.cc:341] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:47.045374 1260 ts_tablet_manager.cc:1397] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: Time spent bootstrapping tablet: real 0.065s user 0.017s sys 0.009s
I20260927 23:29:47.044597 1264 raft_consensus.cc:348] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.046504 1264 raft_consensus.cc:374] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:47.046790 1264 raft_consensus.cc:729] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Initialized, Role: FOLLOWER
I20260927 23:29:47.037210 1310 raft_consensus.cc:686] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.047777 1268 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:47.047576 1264 consensus_queue.cc:260] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.048930 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:47.049532 1145 raft_consensus.cc:2454] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 0.
I20260927 23:29:47.049741 1264 ts_tablet_manager.cc:1428] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20260927 23:29:47.050841 1260 raft_consensus.cc:348] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.051308 1260 raft_consensus.cc:374] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:47.051515 1260 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Initialized, Role: FOLLOWER
I20260927 23:29:47.052703 1184 leader_election.cc:302] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.053396 1301 raft_consensus.cc:2781] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:47.053812 1301 raft_consensus.cc:482] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:47.054157 1301 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.048442 1310 consensus_queue.cc:237] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.051990 1260 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.040647 1291 leader_election.cc:288] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:47.061663 1260 ts_tablet_manager.cc:1428] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260927 23:29:47.048435 1268 raft_consensus.cc:686] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.062249 1301 raft_consensus.cc:504] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.052351 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:47.066250 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.067987 1263 tablet_bootstrap.cc:654] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:47.067888 1145 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.068066 1220 raft_consensus.cc:2379] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e501af30d1b42598487d926ba2719fd in current term 1: Already voted for candidate 789a6d94ecf34e1daf6aa60f4175672b in this term.
I20260927 23:29:47.067621 1301 leader_election.cc:288] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:47.066148 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.074765 967 catalog_manager.cc:5495] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.063730 1268 consensus_queue.cc:237] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.067554 1071 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 1.
I20260927 23:29:47.083050 1035 leader_election.cc:302] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd; no voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b
W20260927 23:29:47.083384 1186 leader_election.cc:341] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:29:47.084666 1310 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.090996 1263 tablet_bootstrap.cc:492] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:47.091434 1263 ts_tablet_manager.cc:1397] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.010s
I20260927 23:29:47.094671 1263 raft_consensus.cc:348] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.095464 1263 raft_consensus.cc:374] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:47.095595 1310 raft_consensus.cc:2726] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:47.095781 1263 raft_consensus.cc:729] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:47.088917 1145 raft_consensus.cc:2454] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 1.
I20260927 23:29:47.095916 965 catalog_manager.cc:5495] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 0 to 1, leader changed from <none> to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 1 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.097157 1263 consensus_queue.cc:260] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.097990 1184 leader_election.cc:302] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.099152 1301 raft_consensus.cc:2781] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:47.099576 1263 ts_tablet_manager.cc:1428] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 23:29:47.099599 1301 raft_consensus.cc:686] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:47.100663 1263 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: Bootstrap starting.
I20260927 23:29:47.100687 1301 consensus_queue.cc:237] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.107569 1263 tablet_bootstrap.cc:654] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: Neither blocks nor log segments found. Creating new log.
I20260927 23:29:47.114841 1263 tablet_bootstrap.cc:492] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: No bootstrap required, opened a new log
I20260927 23:29:47.115237 1263 ts_tablet_manager.cc:1397] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 23:29:47.117151 1263 raft_consensus.cc:348] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.117655 1263 raft_consensus.cc:374] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:29:47.117865 1263 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Initialized, Role: FOLLOWER
I20260927 23:29:47.118404 1263 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.120106 1263 ts_tablet_manager.cc:1428] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 23:29:47.120626 967 catalog_manager.cc:5495] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.135619 966 catalog_manager.cc:5495] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
W20260927 23:29:47.167209 1172 tablet.cc:2369] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:29:47.200904 1310 raft_consensus.cc:482] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.201406 1310 raft_consensus.cc:504] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.203195 1310 leader_election.cc:288] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.203912 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:47.204339 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:47.204766 1145 raft_consensus.cc:2379] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e501af30d1b42598487d926ba2719fd in current term 1: Already voted for candidate 35739abf9980475798a7457f7dfac761 in this term.
I20260927 23:29:47.205874 1035 leader_election.cc:302] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd; no voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.206826 1310 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.211663 1310 raft_consensus.cc:2726] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:47.222446 1310 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.222908 1310 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.224380 1310 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.225286 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:47.225463 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:47.225793 1145 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 0.
I20260927 23:29:47.226060 1220 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 0.
I20260927 23:29:47.226704 1035 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.227331 1310 raft_consensus.cc:2781] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:29:47.227645 1310 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:29:47.227931 1310 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.232874 1310 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.234560 1310 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.235522 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.235750 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.236081 1145 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.236310 1220 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.242115 1145 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 1.
I20260927 23:29:47.242254 1220 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 1.
I20260927 23:29:47.243330 1035 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.244146 1310 raft_consensus.cc:2781] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:29:47.244630 1310 raft_consensus.cc:686] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.245431 1310 consensus_queue.cc:237] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.253238 966 catalog_manager.cc:5495] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 1 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.288545 1268 raft_consensus.cc:482] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.289144 1268 raft_consensus.cc:504] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.290792 1268 leader_election.cc:288] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.291754 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:47.291792 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e501af30d1b42598487d926ba2719fd" is_pre_election: true
I20260927 23:29:47.292448 1220 raft_consensus.cc:2379] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 789a6d94ecf34e1daf6aa60f4175672b in current term 1: Already voted for candidate 1e501af30d1b42598487d926ba2719fd in this term.
I20260927 23:29:47.293300 1110 leader_election.cc:302] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789a6d94ecf34e1daf6aa60f4175672b; no voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761
I20260927 23:29:47.293972 1268 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.298790 1268 raft_consensus.cc:2726] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:47.301764 1301 consensus_queue.cc:1035] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.305442 1303 consensus_queue.cc:1035] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.319330 1291 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.319545 1268 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.338860 1271 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.340834 1304 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.363641 1310 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:29:47.364058 1310 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.365785 1310 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.366452 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b" is_pre_election: true
I20260927 23:29:47.366813 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35739abf9980475798a7457f7dfac761" is_pre_election: true
I20260927 23:29:47.367198 1146 raft_consensus.cc:2379] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e501af30d1b42598487d926ba2719fd in current term 1: Already voted for candidate 35739abf9980475798a7457f7dfac761 in this term.
I20260927 23:29:47.368136 1035 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd; no voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.368744 1310 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:29:47.373154 1310 raft_consensus.cc:2726] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:29:47.451519 1291 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.461719 1301 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.462148 1310 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:47.475641 1310 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.492286 1310 consensus_queue.cc:1035] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.497172 1301 consensus_queue.cc:1035] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.500584 1303 consensus_queue.cc:1035] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.502754 1303 consensus_queue.cc:1035] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:47.512892 1310 consensus_queue.cc:1035] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:47.516117 1219 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:29:47.516907 1219 raft_consensus.cc:595] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.520197 1303 consensus_queue.cc:1035] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:47.524222 1219 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "31555b5cc778446e93141a07ead471e5"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:29:47.524952 1219 raft_consensus.cc:595] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.550266 1262 raft_consensus.cc:979] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.550724 1262 raft_consensus.cc:1067] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.552416 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "31555b5cc778446e93141a07ead471e5"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40310
I20260927 23:29:47.552976 1070 raft_consensus.cc:482] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.553318 1070 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.565461 1145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:60598
I20260927 23:29:47.566092 1145 raft_consensus.cc:595] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.571995 1291 raft_consensus.cc:979] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.572420 1291 raft_consensus.cc:1067] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.573694 1145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:60598
I20260927 23:29:47.574286 1145 raft_consensus.cc:595] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.574815 1068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40310
I20260927 23:29:47.578510 1068 raft_consensus.cc:482] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.578891 1068 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.580700 1070 raft_consensus.cc:504] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.581797 1069 raft_consensus.cc:1226] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Rejecting Update request from peer 35739abf9980475798a7457f7dfac761 for earlier term 1. Current term is 2. Ops: []
I20260927 23:29:47.581756 1220 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:29:47.582472 1220 raft_consensus.cc:595] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:29:47.584573 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.585160 1219 raft_consensus.cc:3032] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.585464 1219 raft_consensus.cc:729] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:47.585935 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31555b5cc778446e93141a07ead471e5" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.586524 1145 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.586208 1219 consensus_queue.cc:260] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.586222 1068 raft_consensus.cc:504] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:47.593068 1068 leader_election.cc:288] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Requested vote from peers 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:47.587412 1219 raft_consensus.cc:3037] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.593948 1145 raft_consensus.cc:2454] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
W20260927 23:29:47.587591 1304 consensus_queue.cc:1159] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1e501af30d1b42598487d926ba2719fd" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 35739abf9980475798a7457f7dfac761 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 23:29:47.599295 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.599892 1220 raft_consensus.cc:3032] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.600188 1220 raft_consensus.cc:729] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:47.601552 1219 raft_consensus.cc:2454] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.601238 1220 consensus_queue.cc:260] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:47.602742 1220 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.588222 1070 leader_election.cc:288] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.618175 1035 leader_election.cc:302] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.619381 1220 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.619731 1310 raft_consensus.cc:2781] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.620193 1310 raft_consensus.cc:686] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.621030 1310 consensus_queue.cc:237] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.621896 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8313caee40fa4473a5c34cbf0f6e0e2b" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.622462 1145 raft_consensus.cc:3037] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.623278 1035 leader_election.cc:302] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:29:47.624015 1309 raft_consensus.cc:2781] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election won for term 2
W20260927 23:29:47.628165 1000 debug-util.cc:398] Leaking SignalData structure 0x7b0800053c40 after lost signal to thread 399
W20260927 23:29:47.629072 1000 debug-util.cc:398] Leaking SignalData structure 0x7b080004b320 after lost signal to thread 1003
I20260927 23:29:47.629073 1145 raft_consensus.cc:2454] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.631358 1271 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.630986 1309 raft_consensus.cc:686] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.633404 1309 consensus_queue.cc:237] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } }
I20260927 23:29:47.641810 1068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:40314
I20260927 23:29:47.642432 1068 raft_consensus.cc:595] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.643098 1311 raft_consensus.cc:979] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.643500 1311 raft_consensus.cc:1067] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.645249 968 catalog_manager.cc:5495] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 1 to 2, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.647254 968 catalog_manager.cc:5495] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 1 to 2, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.649701 1069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40302
I20260927 23:29:47.649688 1311 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:29:47.650409 1069 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.650805 1069 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.654392 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:40314
I20260927 23:29:47.655282 1070 raft_consensus.cc:595] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.659654 1069 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.661553 1069 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.662258 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.662885 1145 raft_consensus.cc:3032] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.663218 1145 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.663522 1218 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:29:47.664155 1218 raft_consensus.cc:595] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.662529 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.664175 1145 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.665958 1145 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.672358 1309 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.675509 1069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "892b90646ced48509346c4d155d805e8"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:40314
I20260927 23:29:47.675828 1220 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.676105 1069 raft_consensus.cc:595] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:29:47.683702 1145 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.684767 1035 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.685238 1220 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.685513 1309 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.686008 1309 raft_consensus.cc:686] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.690040 1309 consensus_queue.cc:237] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.692708 1310 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:29:47.700191 968 catalog_manager.cc:5495] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 1 to 2, leader changed from 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.708451 1309 raft_consensus.cc:979] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.709028 1352 raft_consensus.cc:1067] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.714103 1145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60584
I20260927 23:29:47.714690 1145 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.715070 1145 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.721774 1145 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.723693 1145 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.724794 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.725406 1069 raft_consensus.cc:3032] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.725251 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.725713 1069 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.725888 1220 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.726522 1069 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.728215 1069 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.733861 1220 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.735445 1069 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.737149 1112 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.738852 1311 raft_consensus.cc:2781] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.739355 1311 raft_consensus.cc:686] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.740163 1311 consensus_queue.cc:237] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.752187 966 catalog_manager.cc:5495] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 1 to 2, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 2 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.760383 1301 raft_consensus.cc:979] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.760841 1301 raft_consensus.cc:1067] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.762548 1145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60574
I20260927 23:29:47.763154 1145 raft_consensus.cc:482] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.763976 1145 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.771258 1145 raft_consensus.cc:504] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.773684 1145 leader_election.cc:288] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.774683 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.775228 1220 raft_consensus.cc:3032] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.775228 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.775724 1220 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:47.775921 1069 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.776587 1220 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.777796 1220 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.783067 1069 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.784241 1112 leader_election.cc:302] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.784534 1220 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.785240 1354 raft_consensus.cc:2781] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.785744 1354 raft_consensus.cc:686] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.786617 1354 consensus_queue.cc:237] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.793565 966 catalog_manager.cc:5495] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 1 to 2, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 2 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.900763 1271 raft_consensus.cc:979] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.901247 1271 raft_consensus.cc:1067] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:47.903126 1069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40302
I20260927 23:29:47.903673 1069 raft_consensus.cc:482] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.903972 1069 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.908584 1069 raft_consensus.cc:504] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.911206 1069 leader_election.cc:288] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.911232 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:47.912017 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.912341 1145 raft_consensus.cc:3032] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.912644 1145 raft_consensus.cc:729] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.912684 1220 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.913312 1145 consensus_queue.cc:260] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.914188 1145 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.919756 1145 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.920897 1352 raft_consensus.cc:979] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.920806 1035 leader_election.cc:302] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.921352 1352 raft_consensus.cc:1067] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.922024 1348 raft_consensus.cc:2781] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.922523 1348 raft_consensus.cc:686] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.923360 1145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60584
I20260927 23:29:47.923960 1145 raft_consensus.cc:482] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.923564 1348 consensus_queue.cc:237] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.924321 1145 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.924863 1220 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:47.930263 966 catalog_manager.cc:5495] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 1 to 2, leader changed from 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.936480 1145 raft_consensus.cc:504] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.939242 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.939759 1069 raft_consensus.cc:3032] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.939895 1145 leader_election.cc:288] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.940120 1069 raft_consensus.cc:729] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.940202 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.940973 1220 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.941080 1069 consensus_queue.cc:260] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.942200 1069 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.948093 1220 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.949252 1110 leader_election.cc:302] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.950093 1354 raft_consensus.cc:2781] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.950534 1352 raft_consensus.cc:979] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.950589 1354 raft_consensus.cc:686] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.951036 1352 raft_consensus.cc:1067] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:29:47.951468 1354 consensus_queue.cc:237] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.952242 1069 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.953823 1145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "892b90646ced48509346c4d155d805e8"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60584
I20260927 23:29:47.957087 1145 raft_consensus.cc:482] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:47.957499 1145 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.959481 966 catalog_manager.cc:5495] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 1 to 2, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 2 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:47.963721 1145 raft_consensus.cc:504] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.966310 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:47.966954 1069 raft_consensus.cc:3032] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:47.966845 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892b90646ced48509346c4d155d805e8" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:47.967329 1145 leader_election.cc:288] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:47.967527 1220 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.967338 1069 raft_consensus.cc:729] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:47.969244 1069 consensus_queue.cc:260] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.970486 1069 raft_consensus.cc:3037] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:47.975142 1069 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.975147 1220 raft_consensus.cc:2454] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 2.
I20260927 23:29:47.976300 1112 leader_election.cc:302] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:47.977089 1354 raft_consensus.cc:2781] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:47.977573 1354 raft_consensus.cc:686] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:29:47.978394 1354 consensus_queue.cc:237] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:47.985474 966 catalog_manager.cc:5495] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 1 to 2, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 2 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:48.034222 1145 raft_consensus.cc:1261] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.035895 1348 consensus_queue.cc:1035] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.046000 1220 raft_consensus.cc:1261] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.048112 1352 consensus_queue.cc:1035] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.076416 1301 raft_consensus.cc:979] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:48.076881 1304 raft_consensus.cc:1067] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:29:48.078406 1069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40310
I20260927 23:29:48.078984 1069 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:48.079347 1069 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:48.086700 1069 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:48.088691 1069 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:29:48.089526 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:48.090112 1145 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:48.090052 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:29:48.090723 1220 raft_consensus.cc:3032] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:29:48.091048 1220 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:48.091813 1220 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:48.092990 1220 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:29:48.096750 1145 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:48.097863 1035 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:48.098670 1348 raft_consensus.cc:2781] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:29:48.099150 1348 raft_consensus.cc:686] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:48.099417 1220 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 2.
I20260927 23:29:48.100610 1348 consensus_queue.cc:237] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:48.107738 966 catalog_manager.cc:5495] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 1 to 2, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 2 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:29:48.139473 1220 raft_consensus.cc:1261] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.140717 1348 consensus_queue.cc:1035] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.148772 1145 raft_consensus.cc:1261] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.150168 1309 consensus_queue.cc:1035] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.162930 1145 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.164141 1348 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.178205 1220 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.179366 1310 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.236941 1069 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.238289 1354 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.244562 1220 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.245872 1070 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.246480 1268 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.247671 1268 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.250777 1220 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.261067 1354 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.352337 1070 raft_consensus.cc:1261] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.353610 1354 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.361634 1218 raft_consensus.cc:1261] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.363165 1354 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.505379 1145 raft_consensus.cc:1261] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.506569 1310 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.515275 1218 raft_consensus.cc:1261] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.517048 1352 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.556504 1218 raft_consensus.cc:1261] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.557749 1354 consensus_queue.cc:1035] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.565263 1070 raft_consensus.cc:1261] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.567296 1268 consensus_queue.cc:1035] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.570691 1145 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.572518 1352 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:48.588881 1218 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:29:48.592180 1309 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:29:50.713405 396 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260927 23:29:50.714103 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:50.733670 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:29:50.734261 1070 raft_consensus.cc:595] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Received request to transfer leadership to TS 35739abf9980475798a7457f7dfac761
I20260927 23:29:50.736634 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:29:50.737200 1070 raft_consensus.cc:595] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Received request to transfer leadership to TS 35739abf9980475798a7457f7dfac761
I20260927 23:29:50.738168 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260927 23:29:50.738677 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:50.837641 1361 raft_consensus.cc:979] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:29:50.838147 1362 raft_consensus.cc:1067] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Signalling peer 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:29:50.839689 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:41216
I20260927 23:29:50.840178 1218 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:50.840533 1218 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:50.845305 1218 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:50.847597 1218 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:50.847580 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:50.847805 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:50.848451 1069 raft_consensus.cc:3032] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:50.848572 1146 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:50.848824 1069 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:50.849666 1069 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:50.853590 1146 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:29:50.854568 1069 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:50.856367 1184 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:50.857913 1304 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:50.859143 1304 raft_consensus.cc:686] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:50.859984 1304 consensus_queue.cc:237] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:50.862102 1069 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:29:50.868726 969 catalog_manager.cc:5495] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 2 to 3, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:50.997886 1361 raft_consensus.cc:979] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:29:50.998378 1362 raft_consensus.cc:1067] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Signalling peer 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:29:51.000069 1218 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:41216
I20260927 23:29:51.000725 1218 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:29:51.001099 1218 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:51.008297 1218 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:51.010344 1218 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:29:51.011188 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:29:51.011430 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:29:51.011811 1069 raft_consensus.cc:3032] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:29:51.012023 1146 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:51.012174 1069 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:29:51.012940 1069 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:51.015897 1069 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:29:51.019387 1146 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:29:51.020385 1184 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:29:51.021179 1304 raft_consensus.cc:2781] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:29:51.021714 1304 raft_consensus.cc:686] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:29:51.021078 1069 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:29:51.023116 1304 consensus_queue.cc:237] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:29:51.035212 969 catalog_manager.cc:5495] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 2 to 3, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:29:51.251565 1146 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:51.252705 1304 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:51.261855 1069 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:51.263698 1304 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:51.509420 1146 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:51.510599 1304 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:51.520548 1070 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:29:51.521979 1379 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:29:51.739542 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:51.744984 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:51.745343 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:52.746084 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:52.752645 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:52.753063 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:54.186151 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:54.192516 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:54.192922 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260927 23:29:54.307042 1167 debug-util.cc:398] Leaking SignalData structure 0x7b0800104ac0 after lost signal to thread 399
W20260927 23:29:54.307785 1167 debug-util.cc:398] Leaking SignalData structure 0x7b08001081a0 after lost signal to thread 1003
W20260927 23:29:54.308624 1167 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0c60 after lost signal to thread 1095
W20260927 23:29:54.309649 1167 debug-util.cc:398] Leaking SignalData structure 0x7b08001521e0 after lost signal to thread 1170
W20260927 23:29:54.310508 1167 debug-util.cc:398] Leaking SignalData structure 0x7b08001bdf40 after lost signal to thread 1244
I20260927 23:29:55.193944 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:55.200616 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:55.201022 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:56.201798 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:56.207026 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:56.207430 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:57.208225 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:57.214735 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:57.215238 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:58.216104 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:58.221132 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:58.221479 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:29:59.222280 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:29:59.227885 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:29:59.228329 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:00.229219 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:00.234450 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:00.234792 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:01.265221 396 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260927 23:30:01.277216 1220 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:30:01.277935 1220 raft_consensus.cc:595] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:30:01.282939 1220 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "35739abf9980475798a7457f7dfac761"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:41222
I20260927 23:30:01.283389 1220 raft_consensus.cc:595] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:30:01.288169 1145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:60598
I20260927 23:30:01.288651 1145 raft_consensus.cc:595] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:30:01.293330 1145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:60598
I20260927 23:30:01.293813 1145 raft_consensus.cc:595] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:30:01.298357 1145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
mode: GRACEFUL
new_leader_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:60598
I20260927 23:30:01.298841 1145 raft_consensus.cc:595] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Received request to transfer leadership to TS 1e501af30d1b42598487d926ba2719fd
I20260927 23:30:01.526075 1395 raft_consensus.cc:979] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.526659 1386 raft_consensus.cc:1067] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.528118 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40310
I20260927 23:30:01.528641 1070 raft_consensus.cc:482] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:01.528990 1070 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.534734 1070 raft_consensus.cc:504] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.536433 1070 leader_election.cc:288] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 4 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:01.537354 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:01.537456 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d5cee9e4fc45c790038239db69baf4" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:01.538085 1145 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.538146 1220 raft_consensus.cc:3032] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:30:01.538514 1220 raft_consensus.cc:729] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:30:01.539281 1220 consensus_queue.cc:260] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.540246 1220 raft_consensus.cc:3037] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.543028 1145 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 4.
I20260927 23:30:01.544104 1035 leader_election.cc:302] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:01.544785 1361 raft_consensus.cc:2781] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:30:01.545320 1220 raft_consensus.cc:2454] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 4.
I20260927 23:30:01.545709 1361 raft_consensus.cc:686] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:01.546854 1361 consensus_queue.cc:237] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.551211 1398 raft_consensus.cc:979] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.552006 1392 raft_consensus.cc:1067] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.553705 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40302
I20260927 23:30:01.554190 1070 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:01.554546 1070 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.562265 1070 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.561866 966 catalog_manager.cc:5495] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 3 to 4, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 4 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:01.564255 1070 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:01.565092 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:01.565673 1145 raft_consensus.cc:3032] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:01.565546 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:01.565982 1145 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:30:01.566190 1220 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.566712 1145 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.567962 1145 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.572788 1220 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.573791 1035 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:30:01.574353 1145 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.574653 1361 raft_consensus.cc:2781] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:01.575233 1361 raft_consensus.cc:686] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:01.576112 1361 consensus_queue.cc:237] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.584625 966 catalog_manager.cc:5495] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 2 to 3, leader changed from 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 3 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:01.594986 1398 raft_consensus.cc:979] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.595489 1392 raft_consensus.cc:1067] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.596905 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40302
I20260927 23:30:01.597361 1070 raft_consensus.cc:482] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:01.597606 1070 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.601662 1070 raft_consensus.cc:504] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.602911 1070 leader_election.cc:288] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:01.603829 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:01.604283 1145 raft_consensus.cc:3032] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:01.604077 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1cfd87a16a4bdd97a15922ebd50465" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:01.604537 1145 raft_consensus.cc:729] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:30:01.604754 1220 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.605278 1145 consensus_queue.cc:260] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.606527 1145 raft_consensus.cc:3037] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.609956 1220 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.610912 1035 leader_election.cc:302] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:30:01.611205 1145 raft_consensus.cc:2454] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.611518 1361 raft_consensus.cc:2781] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:01.611920 1361 raft_consensus.cc:686] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:01.612974 1361 consensus_queue.cc:237] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.619549 966 catalog_manager.cc:5495] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 2 to 3, leader changed from 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 3 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:01.635908 1395 raft_consensus.cc:979] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.636315 1386 raft_consensus.cc:1067] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.637724 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40310
I20260927 23:30:01.638152 1070 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:01.638437 1070 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.642390 1070 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.643663 1070 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 4 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:01.644701 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:01.644680 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:01.645305 1220 raft_consensus.cc:3032] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:30:01.645588 1145 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.645644 1220 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:30:01.646107 1398 raft_consensus.cc:979] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: : Instructing follower 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.646600 1392 raft_consensus.cc:1067] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Signalling peer 1e501af30d1b42598487d926ba2719fd to start an election
I20260927 23:30:01.646605 1220 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.647959 1220 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:01.648149 1070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
from {username='slave'} at 127.0.0.1:40302
I20260927 23:30:01.648679 1070 raft_consensus.cc:482] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:01.649047 1070 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.651461 1145 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 4.
I20260927 23:30:01.652436 1220 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 4.
I20260927 23:30:01.652431 1035 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:01.653277 1361 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:30:01.653668 1361 raft_consensus.cc:686] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:01.654461 1361 consensus_queue.cc:237] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.654644 1070 raft_consensus.cc:504] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.656109 1070 leader_election.cc:288] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Requested vote from peers 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:01.657269 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:01.657418 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "1e501af30d1b42598487d926ba2719fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:01.657816 1145 raft_consensus.cc:3032] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:01.657899 1220 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.658152 1145 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:30:01.658897 1145 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.660013 1145 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:01.664229 1220 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.665230 1035 leader_election.cc:302] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 35739abf9980475798a7457f7dfac761; no voters:
I20260927 23:30:01.665998 1361 raft_consensus.cc:2781] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:01.666414 1361 raft_consensus.cc:686] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Becoming Leader. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:01.667111 1361 consensus_queue.cc:237] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:01.667547 1145 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e501af30d1b42598487d926ba2719fd in term 3.
I20260927 23:30:01.667748 966 catalog_manager.cc:5495] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 3 to 4, leader changed from 35739abf9980475798a7457f7dfac761 (127.0.99.3) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 4 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:01.678975 966 catalog_manager.cc:5495] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd reported cstate change: term changed from 2 to 3, leader changed from 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2) to 1e501af30d1b42598487d926ba2719fd (127.0.99.1). New cstate: current_term: 3 leader_uuid: "1e501af30d1b42598487d926ba2719fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:01.957253 1220 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:01.958590 1361 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:01.961054 1145 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:01.962591 1396 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:01.988332 1220 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:01.989512 1410 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:01.998476 1145 raft_consensus.cc:1261] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:02.000056 1396 consensus_queue.cc:1035] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:02.075085 1220 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:02.076309 1410 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:02.085613 1145 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:02.086922 1410 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:02.116856 1220 raft_consensus.cc:1261] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:02.118280 1410 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:02.126819 1145 raft_consensus.cc:1261] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:02.128502 1396 consensus_queue.cc:1035] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:02.236758 1145 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:02.238052 1396 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:02.246757 1220 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Refusing update from remote peer 1e501af30d1b42598487d926ba2719fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:02.248517 1410 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:04.333304 396 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260927 23:30:04.333999 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:04.341686 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:30:04.342211 1070 raft_consensus.cc:595] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:30:04.344743 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:30:04.345268 1070 raft_consensus.cc:595] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Received request to transfer leadership to TS 35739abf9980475798a7457f7dfac761
I20260927 23:30:04.347167 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:30:04.347556 1070 raft_consensus.cc:595] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Received request to transfer leadership to TS 789a6d94ecf34e1daf6aa60f4175672b
I20260927 23:30:04.349366 1070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "1e501af30d1b42598487d926ba2719fd"
mode: GRACEFUL
new_leader_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:56596
I20260927 23:30:04.349767 1070 raft_consensus.cc:595] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Received request to transfer leadership to TS 35739abf9980475798a7457f7dfac761
I20260927 23:30:04.350672 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 23:30:04.351200 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:04.538203 1361 raft_consensus.cc:979] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:30:04.538908 1361 raft_consensus.cc:1067] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Signalling peer 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:30:04.540596 1220 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911"
dest_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:41216
I20260927 23:30:04.541247 1220 raft_consensus.cc:482] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:04.541517 1220 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.546886 1396 raft_consensus.cc:979] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:30:04.547612 1361 raft_consensus.cc:1067] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:30:04.549443 1145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60584
I20260927 23:30:04.549983 1145 raft_consensus.cc:482] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:04.550287 1145 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.551561 1220 raft_consensus.cc:504] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.554834 1220 leader_election.cc:288] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 4 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:30:04.556411 1070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:30:04.556648 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9a9dc40e674fd6a5430bfd0aa9f911" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:04.557147 1070 raft_consensus.cc:3032] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:30:04.557407 1146 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.557514 1070 raft_consensus.cc:729] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:04.558532 1070 consensus_queue.cc:260] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.559137 1145 raft_consensus.cc:504] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.559957 1070 raft_consensus.cc:3037] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.562197 1145 leader_election.cc:288] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 4 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:04.562914 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:30:04.563627 1069 raft_consensus.cc:3032] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:30:04.563884 1069 raft_consensus.cc:729] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:04.564670 1069 consensus_queue.cc:260] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.565735 1069 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.566133 1146 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 4.
I20260927 23:30:04.567557 1184 leader_election.cc:302] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:04.568523 1070 raft_consensus.cc:2454] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 4.
I20260927 23:30:04.562930 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0949c6f71ad54c289d0a40b1a057bedd" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:04.569674 1220 raft_consensus.cc:3037] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:04.570021 1386 raft_consensus.cc:2781] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:30:04.572945 1069 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 4.
I20260927 23:30:04.573304 1386 raft_consensus.cc:686] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 4 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:30:04.574430 1220 raft_consensus.cc:2454] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 4.
I20260927 23:30:04.574465 1112 leader_election.cc:302] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e501af30d1b42598487d926ba2719fd, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:04.574615 1386 consensus_queue.cc:237] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.575735 1392 raft_consensus.cc:2781] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:30:04.576759 1392 raft_consensus.cc:686] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 4 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:30:04.577931 1392 consensus_queue.cc:237] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.586900 969 catalog_manager.cc:5495] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 3 to 4, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:04.588477 966 catalog_manager.cc:5495] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 3 to 4, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 4 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:04.605404 1396 raft_consensus.cc:979] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:30:04.606519 1361 raft_consensus.cc:1067] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Signalling peer 35739abf9980475798a7457f7dfac761 to start an election
I20260927 23:30:04.607975 1220 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a"
dest_uuid: "35739abf9980475798a7457f7dfac761"
from {username='slave'} at 127.0.0.1:41216
I20260927 23:30:04.608459 1220 raft_consensus.cc:482] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:04.608758 1220 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:04.613101 1220 raft_consensus.cc:504] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.614420 1220 leader_election.cc:288] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071)
I20260927 23:30:04.615314 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:30:04.615334 1146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654b86b5107e4105a9ab44ecd0e3f13a" candidate_uuid: "35739abf9980475798a7457f7dfac761" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
I20260927 23:30:04.615768 1069 raft_consensus.cc:3032] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:04.615978 1069 raft_consensus.cc:729] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:04.616082 1146 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:04.616606 1069 consensus_queue.cc:260] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.617514 1069 raft_consensus.cc:3037] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:04.620911 1146 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:30:04.621917 1069 raft_consensus.cc:2454] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35739abf9980475798a7457f7dfac761 in term 3.
I20260927 23:30:04.621966 1184 leader_election.cc:302] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:04.622877 1386 raft_consensus.cc:2781] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:04.623312 1386 raft_consensus.cc:686] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Becoming Leader. State: Replica: 35739abf9980475798a7457f7dfac761, State: Running, Role: LEADER
I20260927 23:30:04.624125 1386 consensus_queue.cc:237] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.631434 969 catalog_manager.cc:5495] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 reported cstate change: term changed from 2 to 3, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 35739abf9980475798a7457f7dfac761 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "35739abf9980475798a7457f7dfac761" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:04.721019 1431 raft_consensus.cc:979] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: : Instructing follower 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:30:04.721637 1361 raft_consensus.cc:1067] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Signalling peer 789a6d94ecf34e1daf6aa60f4175672b to start an election
I20260927 23:30:04.723003 1146 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6"
dest_uuid: "789a6d94ecf34e1daf6aa60f4175672b"
from {username='slave'} at 127.0.0.1:60584
I20260927 23:30:04.723498 1146 raft_consensus.cc:482] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:04.723759 1146 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:30:04.728963 1146 raft_consensus.cc:504] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.730696 1146 leader_election.cc:288] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 5 election: Requested vote from peers 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985), 35739abf9980475798a7457f7dfac761 (127.0.99.3:33021)
I20260927 23:30:04.731354 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1e501af30d1b42598487d926ba2719fd"
I20260927 23:30:04.731725 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4ed00c8fdda44fdba9d7e79f076f6d6" candidate_uuid: "789a6d94ecf34e1daf6aa60f4175672b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "35739abf9980475798a7457f7dfac761"
I20260927 23:30:04.731904 1069 raft_consensus.cc:3032] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:30:04.732311 1220 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:30:04.732319 1069 raft_consensus.cc:729] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e501af30d1b42598487d926ba2719fd, State: Running, Role: LEADER
I20260927 23:30:04.733289 1069 consensus_queue.cc:260] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.734558 1069 raft_consensus.cc:3037] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:30:04.736999 1220 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 5.
I20260927 23:30:04.738221 1110 leader_election.cc:302] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35739abf9980475798a7457f7dfac761, 789a6d94ecf34e1daf6aa60f4175672b; no voters:
I20260927 23:30:04.738965 1392 raft_consensus.cc:2781] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:30:04.739261 1069 raft_consensus.cc:2454] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 789a6d94ecf34e1daf6aa60f4175672b in term 5.
I20260927 23:30:04.739445 1392 raft_consensus.cc:686] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 5 LEADER]: Becoming Leader. State: Replica: 789a6d94ecf34e1daf6aa60f4175672b, State: Running, Role: LEADER
I20260927 23:30:04.740453 1392 consensus_queue.cc:237] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } }
I20260927 23:30:04.748204 969 catalog_manager.cc:5495] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b reported cstate change: term changed from 4 to 5, leader changed from 1e501af30d1b42598487d926ba2719fd (127.0.99.1) to 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2). New cstate: current_term: 5 leader_uuid: "789a6d94ecf34e1daf6aa60f4175672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:04.956845 1069 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:04.958024 1392 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:04.968801 1220 raft_consensus.cc:1261] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:04.971508 1392 consensus_queue.cc:1035] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:05.085613 1146 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:05.086856 1386 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:05.098549 1069 raft_consensus.cc:1261] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:30:05.100062 1441 consensus_queue.cc:1035] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:05.177466 1069 raft_consensus.cc:1261] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:05.178664 1441 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:05.186780 1145 raft_consensus.cc:1261] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Refusing update from remote peer 35739abf9980475798a7457f7dfac761: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:05.189407 1441 consensus_queue.cc:1035] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789a6d94ecf34e1daf6aa60f4175672b" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:05.232074 1069 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 5 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:30:05.233230 1392 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e501af30d1b42598487d926ba2719fd" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:05.242915 1220 raft_consensus.cc:1261] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 5 FOLLOWER]: Refusing update from remote peer 789a6d94ecf34e1daf6aa60f4175672b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:30:05.244701 1436 consensus_queue.cc:1035] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35739abf9980475798a7457f7dfac761" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:05.352934 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:05.358060 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:05.358503 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:06.359380 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:06.364456 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:06.364928 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:07.365828 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:07.371846 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:07.372356 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:08.373389 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:08.379175 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:08.379739 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:09.380499 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:09.386194 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:09.386672 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:10.387462 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:10.392925 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:10.393378 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:11.394151 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:11.400377 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:11.400815 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:12.401892 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:12.407315 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:12.407697 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:13.408552 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:13.415575 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 23:30:13.416149 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:14.458380 396 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260927 23:30:14.459415 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20260927 23:30:14.496958 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:30:14.497958 396 tablet_replica.cc:330] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.499635 396 raft_consensus.cc:2229] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.500362 396 raft_consensus.cc:2258] T 0949c6f71ad54c289d0a40b1a057bedd P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.502517 396 tablet_replica.cc:330] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.503080 396 raft_consensus.cc:2229] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.503539 396 raft_consensus.cc:2258] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 1e501af30d1b42598487d926ba2719fd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.505338 396 tablet_replica.cc:330] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.505875 396 raft_consensus.cc:2229] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 4 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.507157 396 raft_consensus.cc:2258] T 59d5cee9e4fc45c790038239db69baf4 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.509500 396 tablet_replica.cc:330] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.510305 396 raft_consensus.cc:2229] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Raft consensus shutting down.
W20260927 23:30:14.511176 1186 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
I20260927 23:30:14.512324 396 raft_consensus.cc:2258] T 8313caee40fa4473a5c34cbf0f6e0e2b P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Raft consensus is shut down!
W20260927 23:30:14.514585 1186 consensus_peers.cc:487] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 -> Peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Couldn't send request to peer 1e501af30d1b42598487d926ba2719fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:14.515051 396 tablet_replica.cc:330] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.515507 396 raft_consensus.cc:2229] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.516288 396 raft_consensus.cc:2258] T 31555b5cc778446e93141a07ead471e5 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.517913 396 tablet_replica.cc:330] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.518402 396 raft_consensus.cc:2229] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.518826 396 raft_consensus.cc:2258] T 892b90646ced48509346c4d155d805e8 P 1e501af30d1b42598487d926ba2719fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.520533 396 tablet_replica.cc:330] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.520994 396 raft_consensus.cc:2229] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.521497 396 raft_consensus.cc:2258] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 1e501af30d1b42598487d926ba2719fd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.523325 396 tablet_replica.cc:330] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.523758 396 raft_consensus.cc:2229] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.524149 396 raft_consensus.cc:2258] T 654b86b5107e4105a9ab44ecd0e3f13a P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.526472 396 tablet_replica.cc:330] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.526986 396 raft_consensus.cc:2229] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.527944 396 raft_consensus.cc:2258] T fa1cfd87a16a4bdd97a15922ebd50465 P 1e501af30d1b42598487d926ba2719fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.529739 396 tablet_replica.cc:330] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd: stopping tablet replica
I20260927 23:30:14.530207 396 raft_consensus.cc:2229] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.530687 396 raft_consensus.cc:2258] T bfd4b4aa3a364d96805f63e9706818f7 P 1e501af30d1b42598487d926ba2719fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.562536 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20260927 23:30:14.583827 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
W20260927 23:30:14.606060 1184 consensus_peers.cc:487] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 -> Peer 789a6d94ecf34e1daf6aa60f4175672b (127.0.99.2:42071): Couldn't send request to peer 789a6d94ecf34e1daf6aa60f4175672b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:14.611515 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:30:14.612588 396 tablet_replica.cc:330] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.613533 396 raft_consensus.cc:2229] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.614161 396 raft_consensus.cc:2258] T bfd4b4aa3a364d96805f63e9706818f7 P 789a6d94ecf34e1daf6aa60f4175672b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 23:30:14.615515 1186 consensus_peers.cc:487] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 -> Peer 1e501af30d1b42598487d926ba2719fd (127.0.99.1:32985): Couldn't send request to peer 1e501af30d1b42598487d926ba2719fd. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.1:32985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:14.616533 396 tablet_replica.cc:330] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.617080 396 raft_consensus.cc:2229] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 5 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.617991 396 raft_consensus.cc:2258] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 789a6d94ecf34e1daf6aa60f4175672b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.619824 396 tablet_replica.cc:330] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.620467 396 raft_consensus.cc:2229] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.621344 396 raft_consensus.cc:2258] T 892b90646ced48509346c4d155d805e8 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.622954 396 tablet_replica.cc:330] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.623499 396 raft_consensus.cc:2229] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.623957 396 raft_consensus.cc:2258] T 31555b5cc778446e93141a07ead471e5 P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.625933 396 tablet_replica.cc:330] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.626500 396 raft_consensus.cc:2229] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.626983 396 raft_consensus.cc:2258] T 59d5cee9e4fc45c790038239db69baf4 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.628525 396 tablet_replica.cc:330] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.629099 396 raft_consensus.cc:2229] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.629788 396 raft_consensus.cc:2258] T 8313caee40fa4473a5c34cbf0f6e0e2b P 789a6d94ecf34e1daf6aa60f4175672b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.631745 396 tablet_replica.cc:330] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.632196 396 raft_consensus.cc:2229] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.632586 396 raft_consensus.cc:2258] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.634208 396 tablet_replica.cc:330] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.634734 396 raft_consensus.cc:2229] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.635179 396 raft_consensus.cc:2258] T fa1cfd87a16a4bdd97a15922ebd50465 P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.636883 396 tablet_replica.cc:330] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.637321 396 raft_consensus.cc:2229] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.637789 396 raft_consensus.cc:2258] T 654b86b5107e4105a9ab44ecd0e3f13a P 789a6d94ecf34e1daf6aa60f4175672b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.639441 396 tablet_replica.cc:330] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b: stopping tablet replica
I20260927 23:30:14.639869 396 raft_consensus.cc:2229] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 4 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.640774 396 raft_consensus.cc:2258] T 0949c6f71ad54c289d0a40b1a057bedd P 789a6d94ecf34e1daf6aa60f4175672b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.684820 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20260927 23:30:14.701622 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20260927 23:30:14.728778 396 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 23:30:14.729537 396 tablet_replica.cc:330] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.730402 396 raft_consensus.cc:2229] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.731154 396 raft_consensus.cc:2258] T 0949c6f71ad54c289d0a40b1a057bedd P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.733068 396 tablet_replica.cc:330] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.733817 396 raft_consensus.cc:2229] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.734400 396 raft_consensus.cc:2258] T b4ed00c8fdda44fdba9d7e79f076f6d6 P 35739abf9980475798a7457f7dfac761 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.736340 396 tablet_replica.cc:330] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.736856 396 raft_consensus.cc:2229] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.737886 396 raft_consensus.cc:2258] T bfd4b4aa3a364d96805f63e9706818f7 P 35739abf9980475798a7457f7dfac761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.739892 396 tablet_replica.cc:330] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.740518 396 raft_consensus.cc:2229] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 3 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.741521 396 raft_consensus.cc:2258] T 654b86b5107e4105a9ab44ecd0e3f13a P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.743453 396 tablet_replica.cc:330] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.744028 396 raft_consensus.cc:2229] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.744544 396 raft_consensus.cc:2258] T fa1cfd87a16a4bdd97a15922ebd50465 P 35739abf9980475798a7457f7dfac761 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.746294 396 tablet_replica.cc:330] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.746887 396 raft_consensus.cc:2229] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.747366 396 raft_consensus.cc:2258] T 8313caee40fa4473a5c34cbf0f6e0e2b P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.749089 396 tablet_replica.cc:330] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.749598 396 raft_consensus.cc:2229] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.750131 396 raft_consensus.cc:2258] T 59d5cee9e4fc45c790038239db69baf4 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.751888 396 tablet_replica.cc:330] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.752386 396 raft_consensus.cc:2229] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.752841 396 raft_consensus.cc:2258] T 31555b5cc778446e93141a07ead471e5 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.754494 396 tablet_replica.cc:330] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.754999 396 raft_consensus.cc:2229] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:14.755443 396 raft_consensus.cc:2258] T 892b90646ced48509346c4d155d805e8 P 35739abf9980475798a7457f7dfac761 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.757094 396 tablet_replica.cc:330] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761: stopping tablet replica
I20260927 23:30:14.757573 396 raft_consensus.cc:2229] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 4 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.758522 396 raft_consensus.cc:2258] T cc9a9dc40e674fd6a5430bfd0aa9f911 P 35739abf9980475798a7457f7dfac761 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.788201 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20260927 23:30:14.807651 396 master.cc:537] Master@127.0.99.62:37253 shutting down...
I20260927 23:30:14.839604 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:30:14.840138 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:14.840536 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1d3c5d2f284d484bbe6a841cc2a782e2: stopping tablet replica
I20260927 23:30:14.881171 396 master.cc:552] Master@127.0.99.62:37253 shutdown complete.
1e501af30d1b42598487d926ba2719fd 5
35739abf9980475798a7457f7dfac761 1
789a6d94ecf34e1daf6aa60f4175672b 4
1e501af30d1b42598487d926ba2719fd 3
35739abf9980475798a7457f7dfac761 3
789a6d94ecf34e1daf6aa60f4175672b 4
1e501af30d1b42598487d926ba2719fd 8
35739abf9980475798a7457f7dfac761 1
789a6d94ecf34e1daf6aa60f4175672b 1
1e501af30d1b42598487d926ba2719fd 4
35739abf9980475798a7457f7dfac761 3
789a6d94ecf34e1daf6aa60f4175672b 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29411 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260927 23:30:14.923408 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:38077
I20260927 23:30:14.924474 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:30:14.930361 1461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:14.930413 1462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:14.931397 1464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:14.932557 396 server_base.cc:1034] running on GCE node
I20260927 23:30:14.933418 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:30:14.933586 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:30:14.933689 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551814933682 us; error 0 us; skew 500 ppm
I20260927 23:30:14.934171 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:30:14.936754 396 webserver.cc:460] Webserver started at http://127.0.99.62:34091/ using document root <none> and password file <none>
I20260927 23:30:14.937256 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:30:14.937433 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:30:14.937739 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:30:14.939109 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/master-0-root/instance:
uuid: "c2f29821eafd45fbbcc1b22680919836"
format_stamp: "Formatted at 2026-09-27 23:30:14 on dist-test-slave-zkzt"
I20260927 23:30:14.943359 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:30:14.946784 1469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:14.947558 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:30:14.947882 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/master-0-root
uuid: "c2f29821eafd45fbbcc1b22680919836"
format_stamp: "Formatted at 2026-09-27 23:30:14 on dist-test-slave-zkzt"
I20260927 23:30:14.948170 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:30:14.958976 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:30:14.960237 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:30:14.997666 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:38077
I20260927 23:30:14.997776 1520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:38077 every 8 connection(s)
I20260927 23:30:15.001379 1521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.010692 1521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Bootstrap starting.
I20260927 23:30:15.014853 1521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:15.019124 1521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: No bootstrap required, opened a new log
I20260927 23:30:15.021343 1521 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER }
I20260927 23:30:15.021833 1521 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:15.022033 1521 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f29821eafd45fbbcc1b22680919836, State: Initialized, Role: FOLLOWER
I20260927 23:30:15.022686 1521 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER }
I20260927 23:30:15.023137 1521 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:30:15.023350 1521 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:30:15.023577 1521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:15.027719 1521 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER }
I20260927 23:30:15.028205 1521 leader_election.cc:302] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f29821eafd45fbbcc1b22680919836; no voters:
I20260927 23:30:15.029488 1521 leader_election.cc:288] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:30:15.030057 1524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:15.031554 1524 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [term 1 LEADER]: Becoming Leader. State: Replica: c2f29821eafd45fbbcc1b22680919836, State: Running, Role: LEADER
I20260927 23:30:15.032207 1524 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER }
I20260927 23:30:15.032626 1521 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:30:15.036972 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f29821eafd45fbbcc1b22680919836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER } }
I20260927 23:30:15.037978 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [sys.catalog]: This master's current role is: LEADER
I20260927 23:30:15.037683 1526 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f29821eafd45fbbcc1b22680919836. Latest consensus state: current_term: 1 leader_uuid: "c2f29821eafd45fbbcc1b22680919836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f29821eafd45fbbcc1b22680919836" member_type: VOTER } }
I20260927 23:30:15.038738 1526 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836 [sys.catalog]: This master's current role is: LEADER
I20260927 23:30:15.041872 1531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:30:15.049913 1531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:30:15.050925 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 23:30:15.058643 1531 catalog_manager.cc:1347] Generated new cluster ID: 47de722cc32d4c3d9062d19e8ca30b30
I20260927 23:30:15.058954 1531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:30:15.085691 1531 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:30:15.087278 1531 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:30:15.097147 1531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Generated new TSK 0
I20260927 23:30:15.097764 1531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:30:15.118057 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:30:15.124306 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:15.125108 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:15.127825 396 server_base.cc:1034] running on GCE node
W20260927 23:30:15.127843 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:15.129053 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:30:15.129264 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:30:15.129439 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551815129425 us; error 0 us; skew 500 ppm
I20260927 23:30:15.129961 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:30:15.132292 396 webserver.cc:460] Webserver started at http://127.0.99.1:37723/ using document root <none> and password file <none>
I20260927 23:30:15.132776 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:30:15.133028 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:30:15.133348 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:30:15.134626 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/instance:
uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.139281 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 23:30:15.142454 1550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.143373 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 23:30:15.143662 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root
uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.143957 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:30:15.156894 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:30:15.158116 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:30:15.159515 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:30:15.161684 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:30:15.161856 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.162153 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:30:15.162314 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.201463 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:45501
I20260927 23:30:15.201576 1612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:45501 every 8 connection(s)
I20260927 23:30:15.206312 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:30:15.214691 1617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:15.217195 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:15.218241 396 server_base.cc:1034] running on GCE node
W20260927 23:30:15.220052 1620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:15.220988 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:30:15.221212 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:30:15.221480 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551815221462 us; error 0 us; skew 500 ppm
I20260927 23:30:15.221976 1613 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38077
I20260927 23:30:15.222071 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:30:15.222477 1613 heartbeater.cc:464] Registering TS with master...
I20260927 23:30:15.223470 1613 heartbeater.cc:511] Master 127.0.99.62:38077 requested a full tablet report, sending...
I20260927 23:30:15.224948 396 webserver.cc:460] Webserver started at http://127.0.99.2:36113/ using document root <none> and password file <none>
I20260927 23:30:15.225507 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:30:15.225718 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:30:15.225970 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:30:15.225920 1486 ts_manager.cc:194] Registered new tserver with Master: 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:15.227372 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/instance:
uuid: "1b945905616b4989853b0a92bfbd6b98"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.227923 1486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48976
I20260927 23:30:15.232184 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:30:15.235579 1625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.236474 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:30:15.236713 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root
uuid: "1b945905616b4989853b0a92bfbd6b98"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.236943 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:30:15.258880 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:30:15.259985 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:30:15.261294 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:30:15.263432 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:30:15.263594 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.263842 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:30:15.264015 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.302641 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:42051
I20260927 23:30:15.302802 1687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:42051 every 8 connection(s)
I20260927 23:30:15.307225 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:30:15.314589 1691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:15.315771 1692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:15.317121 1694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:15.317572 396 server_base.cc:1034] running on GCE node
I20260927 23:30:15.318733 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:30:15.319023 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:30:15.319249 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551815319234 us; error 0 us; skew 500 ppm
I20260927 23:30:15.320079 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:30:15.323488 1688 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38077
I20260927 23:30:15.323501 396 webserver.cc:460] Webserver started at http://127.0.99.3:37509/ using document root <none> and password file <none>
I20260927 23:30:15.323992 1688 heartbeater.cc:464] Registering TS with master...
I20260927 23:30:15.324355 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:30:15.324618 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:30:15.324704 1688 heartbeater.cc:511] Master 127.0.99.62:38077 requested a full tablet report, sending...
I20260927 23:30:15.324921 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:30:15.326676 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/instance:
uuid: "a2a3317e8da742d9aefe4c251abd4566"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.326805 1486 ts_manager.cc:194] Registered new tserver with Master: 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:15.328327 1486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48984
I20260927 23:30:15.331784 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:30:15.335567 1699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.336304 396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:30:15.336611 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root
uuid: "a2a3317e8da742d9aefe4c251abd4566"
format_stamp: "Formatted at 2026-09-27 23:30:15 on dist-test-slave-zkzt"
I20260927 23:30:15.336897 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:30:15.351738 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:30:15.352891 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:30:15.354360 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:30:15.356480 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:30:15.356714 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.356943 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:30:15.357106 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:15.396566 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:35731
I20260927 23:30:15.396672 1761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:35731 every 8 connection(s)
I20260927 23:30:15.411867 1762 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38077
I20260927 23:30:15.412242 1762 heartbeater.cc:464] Registering TS with master...
I20260927 23:30:15.412899 1762 heartbeater.cc:511] Master 127.0.99.62:38077 requested a full tablet report, sending...
I20260927 23:30:15.414772 1486 ts_manager.cc:194] Registered new tserver with Master: a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:15.415199 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015754936s
I20260927 23:30:15.416888 1486 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48998
I20260927 23:30:15.417968 396 test_util.cc:268] Using random seed: 1424552022
I20260927 23:30:15.443188 1486 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49006:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:30:15.449600 1486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:30:15.668144 1652 tablet_service.cc:1467] Processing CreateTablet for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 23:30:15.669143 1727 tablet_service.cc:1467] Processing CreateTablet for tablet 0be2a211b7754af89c38527623e5d967 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 23:30:15.669596 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.670393 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be2a211b7754af89c38527623e5d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.671398 1651 tablet_service.cc:1467] Processing CreateTablet for tablet a9680f1581344928ab0dd3e14eebea73 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 23:30:15.672640 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9680f1581344928ab0dd3e14eebea73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.675426 1650 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe23b5411b441fbd8792ef9dc4e99b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 23:30:15.675489 1725 tablet_service.cc:1467] Processing CreateTablet for tablet a9680f1581344928ab0dd3e14eebea73 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 23:30:15.676733 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe23b5411b441fbd8792ef9dc4e99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.676815 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9680f1581344928ab0dd3e14eebea73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.679545 1646 tablet_service.cc:1467] Processing CreateTablet for tablet ada48ebdd6c1449bb76554223d5569f7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 23:30:15.679605 1649 tablet_service.cc:1467] Processing CreateTablet for tablet b8ceb00d8a73495dbaab22eda5654033 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 23:30:15.681085 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada48ebdd6c1449bb76554223d5569f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.681303 1578 tablet_service.cc:1467] Processing CreateTablet for tablet 0be2a211b7754af89c38527623e5d967 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 23:30:15.681551 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d069fe150057489e8a595e9fe7c32de5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 23:30:15.682580 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be2a211b7754af89c38527623e5d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.683435 1647 tablet_service.cc:1467] Processing CreateTablet for tablet b04c203146534f018c1bdcc2596f0010 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 23:30:15.683910 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ac10d629e97c4945b1f2fb7ba7c1540e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 23:30:15.684618 1577 tablet_service.cc:1467] Processing CreateTablet for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 23:30:15.685375 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 0be2a211b7754af89c38527623e5d967 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 23:30:15.685945 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.686517 1726 tablet_service.cc:1467] Processing CreateTablet for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 23:30:15.688105 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.686156 1645 tablet_service.cc:1467] Processing CreateTablet for tablet adf0ac4f0f334ff6862e45d38f191756 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 23:30:15.690971 1724 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe23b5411b441fbd8792ef9dc4e99b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 23:30:15.692538 1724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe23b5411b441fbd8792ef9dc4e99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.694892 1723 tablet_service.cc:1467] Processing CreateTablet for tablet b8ceb00d8a73495dbaab22eda5654033 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 23:30:15.696252 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ceb00d8a73495dbaab22eda5654033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.699156 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d069fe150057489e8a595e9fe7c32de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.701730 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04c203146534f018c1bdcc2596f0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.703919 1576 tablet_service.cc:1467] Processing CreateTablet for tablet a9680f1581344928ab0dd3e14eebea73 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 23:30:15.704605 1574 tablet_service.cc:1467] Processing CreateTablet for tablet b8ceb00d8a73495dbaab22eda5654033 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 23:30:15.705240 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9680f1581344928ab0dd3e14eebea73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.705838 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ceb00d8a73495dbaab22eda5654033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.706532 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac10d629e97c4945b1f2fb7ba7c1540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.707100 1575 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe23b5411b441fbd8792ef9dc4e99b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 23:30:15.707769 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be2a211b7754af89c38527623e5d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.708307 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe23b5411b441fbd8792ef9dc4e99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.708976 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf0ac4f0f334ff6862e45d38f191756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.711005 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ceb00d8a73495dbaab22eda5654033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.731491 1572 tablet_service.cc:1467] Processing CreateTablet for tablet b04c203146534f018c1bdcc2596f0010 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 23:30:15.733079 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04c203146534f018c1bdcc2596f0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.735401 1571 tablet_service.cc:1467] Processing CreateTablet for tablet ada48ebdd6c1449bb76554223d5569f7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 23:30:15.737138 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada48ebdd6c1449bb76554223d5569f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.739746 1569 tablet_service.cc:1467] Processing CreateTablet for tablet ac10d629e97c4945b1f2fb7ba7c1540e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 23:30:15.741117 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac10d629e97c4945b1f2fb7ba7c1540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.743255 1570 tablet_service.cc:1467] Processing CreateTablet for tablet adf0ac4f0f334ff6862e45d38f191756 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 23:30:15.744827 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf0ac4f0f334ff6862e45d38f191756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.730049 1573 tablet_service.cc:1467] Processing CreateTablet for tablet d069fe150057489e8a595e9fe7c32de5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 23:30:15.747320 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d069fe150057489e8a595e9fe7c32de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.772132 1722 tablet_service.cc:1467] Processing CreateTablet for tablet d069fe150057489e8a595e9fe7c32de5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 23:30:15.774070 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d069fe150057489e8a595e9fe7c32de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.780051 1721 tablet_service.cc:1467] Processing CreateTablet for tablet b04c203146534f018c1bdcc2596f0010 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 23:30:15.781402 1718 tablet_service.cc:1467] Processing CreateTablet for tablet ac10d629e97c4945b1f2fb7ba7c1540e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 23:30:15.781427 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b04c203146534f018c1bdcc2596f0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.783658 1719 tablet_service.cc:1467] Processing CreateTablet for tablet adf0ac4f0f334ff6862e45d38f191756 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 23:30:15.784946 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf0ac4f0f334ff6862e45d38f191756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.782564 1720 tablet_service.cc:1467] Processing CreateTablet for tablet ada48ebdd6c1449bb76554223d5569f7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 23:30:15.787930 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada48ebdd6c1449bb76554223d5569f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.789296 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac10d629e97c4945b1f2fb7ba7c1540e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.833388 1777 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:15.834144 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 61b00bbf7bb549d790e4d375530f3e03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 23:30:15.835536 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b00bbf7bb549d790e4d375530f3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.847589 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa0ad1d9f2349b48e2967755ab789bd (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 23:30:15.850358 1652 tablet_service.cc:1467] Processing CreateTablet for tablet e850f344b805403cb981ec2cf5272ac2 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 23:30:15.851735 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e850f344b805403cb981ec2cf5272ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.866744 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260927 23:30:15.868145 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.879122 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa0ad1d9f2349b48e2967755ab789bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.883658 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7cf5ede5b947f8a58db3da81e353ef (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 23:30:15.890952 1779 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:15.893460 1777 tablet_bootstrap.cc:654] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:15.893703 1649 tablet_service.cc:1467] Processing CreateTablet for tablet cee6b5008b764b60a2808b4af785ef8a (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 23:30:15.896485 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 61b00bbf7bb549d790e4d375530f3e03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 23:30:15.896776 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa0ad1d9f2349b48e2967755ab789bd (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 23:30:15.898082 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b00bbf7bb549d790e4d375530f3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.899569 1721 tablet_service.cc:1467] Processing CreateTablet for tablet e850f344b805403cb981ec2cf5272ac2 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 23:30:15.903909 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa0ad1d9f2349b48e2967755ab789bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.906920 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e850f344b805403cb981ec2cf5272ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.924285 1578 tablet_service.cc:1467] Processing CreateTablet for tablet 61b00bbf7bb549d790e4d375530f3e03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 23:30:15.918959 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7cf5ede5b947f8a58db3da81e353ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.925644 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b00bbf7bb549d790e4d375530f3e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.925429 1727 tablet_service.cc:1467] Processing CreateTablet for tablet cee6b5008b764b60a2808b4af785ef8a (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 23:30:15.926931 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcce6620eee4240b1d1b454ac3ca6eb (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 23:30:15.928131 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcce6620eee4240b1d1b454ac3ca6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.931492 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee6b5008b764b60a2808b4af785ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.927289 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa0ad1d9f2349b48e2967755ab789bd (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 23:30:15.920239 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260927 23:30:15.933300 1778 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:15.954532 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee6b5008b764b60a2808b4af785ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.960466 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.966746 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40ddbf07dc427f9a4f51a71f25d985 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 23:30:15.966900 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7cf5ede5b947f8a58db3da81e353ef (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 23:30:15.969637 1718 tablet_service.cc:1467] Processing CreateTablet for tablet f575009df7d846d997a605fb1e38ad31 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 23:30:15.972899 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 505e560fa0174a5987736c3bb71c7753 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 23:30:15.978945 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505e560fa0174a5987736c3bb71c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.972990 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40ddbf07dc427f9a4f51a71f25d985 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 23:30:15.974115 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 505e560fa0174a5987736c3bb71c7753 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 23:30:15.978014 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575009df7d846d997a605fb1e38ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.980942 1724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40ddbf07dc427f9a4f51a71f25d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.982302 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7cf5ede5b947f8a58db3da81e353ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.986629 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505e560fa0174a5987736c3bb71c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:15.996045 1647 tablet_service.cc:1467] Processing CreateTablet for tablet f575009df7d846d997a605fb1e38ad31 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 23:30:15.996460 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 879c1897331c4c32ade3f828689195c4 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 23:30:15.997103 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa0ad1d9f2349b48e2967755ab789bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.003803 1577 tablet_service.cc:1467] Processing CreateTablet for tablet e850f344b805403cb981ec2cf5272ac2 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 23:30:16.004592 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40ddbf07dc427f9a4f51a71f25d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.015682 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7cf5ede5b947f8a58db3da81e353ef (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 23:30:16.015682 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 879c1897331c4c32ade3f828689195c4 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 23:30:16.026186 1777 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:16.028409 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575009df7d846d997a605fb1e38ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.036383 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 74c5e2f6966148308b5ccecb8d8f4313 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 23:30:16.037731 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c5e2f6966148308b5ccecb8d8f4313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.029424 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7cf5ede5b947f8a58db3da81e353ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.042620 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e850f344b805403cb981ec2cf5272ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.026605 1653 tablet_service.cc:1467] Processing CreateTablet for tablet e382c82189e743b4ace885d63e5e9560 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 23:30:16.043533 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e382c82189e743b4ace885d63e5e9560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.045215 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
W20260927 23:30:16.045734 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:16.047055 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 92d480f46ebe4fafb41af6682200fbb8 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 23:30:16.047170 1569 tablet_service.cc:1467] Processing CreateTablet for tablet cee6b5008b764b60a2808b4af785ef8a (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 23:30:16.048347 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d480f46ebe4fafb41af6682200fbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.049681 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 1101ef7672c345ec8f5509edeb86b553 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260927 23:30:16.051055 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1101ef7672c345ec8f5509edeb86b553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.060911 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c947ede0cb8a48cb90cfb0bcc8ed442b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 23:30:16.061115 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfb0bd166774a248820c1b90412697d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 23:30:16.062175 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947ede0cb8a48cb90cfb0bcc8ed442b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.030455 1777 ts_tablet_manager.cc:1397] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.197s user 0.015s sys 0.006s
I20260927 23:30:16.064882 1778 tablet_bootstrap.cc:654] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.065518 1777 raft_consensus.cc:348] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.066357 1777 raft_consensus.cc:374] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.066646 1777 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.067596 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.067447 1777 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.031430 1719 tablet_service.cc:1467] Processing CreateTablet for tablet e382c82189e743b4ace885d63e5e9560 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 23:30:16.070107 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e382c82189e743b4ace885d63e5e9560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.070956 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879c1897331c4c32ade3f828689195c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.071321 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879c1897331c4c32ade3f828689195c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.030061 1779 tablet_bootstrap.cc:654] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.073904 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee6b5008b764b60a2808b4af785ef8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.082286 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 74c5e2f6966148308b5ccecb8d8f4313 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 23:30:16.083758 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c5e2f6966148308b5ccecb8d8f4313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.089697 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfb0bd166774a248820c1b90412697d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.098116 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 92d480f46ebe4fafb41af6682200fbb8 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 23:30:16.101148 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 5d40ddbf07dc427f9a4f51a71f25d985 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 23:30:16.091470 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 505e560fa0174a5987736c3bb71c7753 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 23:30:16.092856 1688 heartbeater.cc:503] Master 127.0.99.62:38077 was elected leader, sending a full tablet report...
I20260927 23:30:16.102453 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d40ddbf07dc427f9a4f51a71f25d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.093899 1777 ts_tablet_manager.cc:1428] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20260927 23:30:16.107641 1570 tablet_service.cc:1467] Processing CreateTablet for tablet f575009df7d846d997a605fb1e38ad31 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 23:30:16.108595 1777 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:16.109411 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 1101ef7672c345ec8f5509edeb86b553 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260927 23:30:16.109974 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcce6620eee4240b1d1b454ac3ca6eb (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 23:30:16.111189 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcce6620eee4240b1d1b454ac3ca6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.107679 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 505e560fa0174a5987736c3bb71c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.115434 1726 tablet_service.cc:1467] Processing CreateTablet for tablet c947ede0cb8a48cb90cfb0bcc8ed442b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 23:30:16.120759 1727 tablet_service.cc:1467] Processing CreateTablet for tablet 4912c6b1a48e4d9087c4b72bef319110 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 23:30:16.121407 1779 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:16.121923 1779 ts_tablet_manager.cc:1397] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.231s user 0.009s sys 0.007s
I20260927 23:30:16.125921 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfb0bd166774a248820c1b90412697d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 23:30:16.136863 1778 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:16.137530 1778 ts_tablet_manager.cc:1397] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.205s user 0.000s sys 0.013s
I20260927 23:30:16.137785 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575009df7d846d997a605fb1e38ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.131397 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 879c1897331c4c32ade3f828689195c4 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 23:30:16.136862 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d480f46ebe4fafb41af6682200fbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.141683 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879c1897331c4c32ade3f828689195c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.142616 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4912c6b1a48e4d9087c4b72bef319110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.145164 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 2bac3d0a2c8444f79b96925750c29b42 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 23:30:16.146165 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947ede0cb8a48cb90cfb0bcc8ed442b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.146561 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bac3d0a2c8444f79b96925750c29b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.152814 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 4912c6b1a48e4d9087c4b72bef319110 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 23:30:16.154063 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4912c6b1a48e4d9087c4b72bef319110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.154640 1777 tablet_bootstrap.cc:654] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.155997 1779 raft_consensus.cc:348] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.156741 1779 raft_consensus.cc:374] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.157018 1779 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.131387 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 5bcce6620eee4240b1d1b454ac3ca6eb (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 23:30:16.158666 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 0803b6723b37453ca7518fb7ced3081d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 23:30:16.158859 1779 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.154654 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1101ef7672c345ec8f5509edeb86b553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.160715 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bcce6620eee4240b1d1b454ac3ca6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.162295 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 2bac3d0a2c8444f79b96925750c29b42 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 23:30:16.163533 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bac3d0a2c8444f79b96925750c29b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.166095 1724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfb0bd166774a248820c1b90412697d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.167405 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 3565b68068904064a965cd0a82a20802 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 23:30:16.173280 1578 tablet_service.cc:1467] Processing CreateTablet for tablet e382c82189e743b4ace885d63e5e9560 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 23:30:16.140375 1778 raft_consensus.cc:348] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.174256 1778 raft_consensus.cc:374] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.175120 1778 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.185808 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 74c5e2f6966148308b5ccecb8d8f4313 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 23:30:16.184373 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e382c82189e743b4ace885d63e5e9560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.189056 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0803b6723b37453ca7518fb7ced3081d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.192421 1778 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.198297 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 1101ef7672c345ec8f5509edeb86b553 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260927 23:30:16.200131 1779 ts_tablet_manager.cc:1428] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.078s user 0.006s sys 0.010s
I20260927 23:30:16.200985 1762 heartbeater.cc:503] Master 127.0.99.62:38077 was elected leader, sending a full tablet report...
I20260927 23:30:16.207592 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c5e2f6966148308b5ccecb8d8f4313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.208845 1577 tablet_service.cc:1467] Processing CreateTablet for tablet c947ede0cb8a48cb90cfb0bcc8ed442b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 23:30:16.205402 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 0803b6723b37453ca7518fb7ced3081d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 23:30:16.212700 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565b68068904064a965cd0a82a20802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.213299 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0803b6723b37453ca7518fb7ced3081d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.214844 1777 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:16.215471 1777 ts_tablet_manager.cc:1397] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.003s
I20260927 23:30:16.216203 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 07438388789747e68da9846e153a903b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 23:30:16.217796 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07438388789747e68da9846e153a903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.220556 1779 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:16.201956 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 2b90254fab234e29ba132c165c446490 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 23:30:16.230907 1613 heartbeater.cc:503] Master 127.0.99.62:38077 was elected leader, sending a full tablet report...
I20260927 23:30:16.236572 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 58cf4a7fdcb146b0a364fa00e849e902 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 23:30:16.240696 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfb0bd166774a248820c1b90412697d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 23:30:16.242484 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 92d480f46ebe4fafb41af6682200fbb8 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 23:30:16.243579 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 502de1f57afe4b808a200ece197c19e6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 23:30:16.253021 1777 raft_consensus.cc:348] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.253816 1777 raft_consensus.cc:374] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.256456 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c947ede0cb8a48cb90cfb0bcc8ed442b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.259478 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1101ef7672c345ec8f5509edeb86b553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.263494 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 3565b68068904064a965cd0a82a20802 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 23:30:16.264747 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565b68068904064a965cd0a82a20802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.266640 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b90254fab234e29ba132c165c446490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.267621 1779 tablet_bootstrap.cc:654] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.267328 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 2bac3d0a2c8444f79b96925750c29b42 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 23:30:16.268570 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bac3d0a2c8444f79b96925750c29b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.270819 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cf4a7fdcb146b0a364fa00e849e902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.274610 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d480f46ebe4fafb41af6682200fbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.278824 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502de1f57afe4b808a200ece197c19e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.278836 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfb0bd166774a248820c1b90412697d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.279531 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 99c7c13b00be47d38145b16ad8a788cc (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 23:30:16.280774 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c7c13b00be47d38145b16ad8a788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.281762 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 2b90254fab234e29ba132c165c446490 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 23:30:16.284773 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 4912c6b1a48e4d9087c4b72bef319110 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 23:30:16.286146 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4912c6b1a48e4d9087c4b72bef319110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.284003 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 07438388789747e68da9846e153a903b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 23:30:16.286270 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b90254fab234e29ba132c165c446490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.304502 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcc903f42c448e1beb437468961e2b0 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 23:30:16.305037 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07438388789747e68da9846e153a903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.305800 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcc903f42c448e1beb437468961e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.312935 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddb51781fec46b194174de5f4406def (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 23:30:16.254103 1777 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.314911 1777 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.317289 1777 ts_tablet_manager.cc:1428] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.101s user 0.007s sys 0.001s
I20260927 23:30:16.318210 1777 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:16.314229 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddb51781fec46b194174de5f4406def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.326130 1781 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:16.334141 1781 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.325297 1778 ts_tablet_manager.cc:1428] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.187s user 0.006s sys 0.009s
I20260927 23:30:16.344873 1778 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:16.315462 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fafc0b4640efa0d437d468e2bdac (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 23:30:16.322554 1727 tablet_service.cc:1467] Processing CreateTablet for tablet 58cf4a7fdcb146b0a364fa00e849e902 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 23:30:16.327430 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 0803b6723b37453ca7518fb7ced3081d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 23:30:16.382053 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cf4a7fdcb146b0a364fa00e849e902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.379580 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 502de1f57afe4b808a200ece197c19e6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 23:30:16.384482 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502de1f57afe4b808a200ece197c19e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.385171 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9629fafc0b4640efa0d437d468e2bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.388828 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 2b90254fab234e29ba132c165c446490 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 23:30:16.389493 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8c92025b82e6497ba49fb88677061e6e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260927 23:30:16.390085 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b90254fab234e29ba132c165c446490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.390825 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 99c7c13b00be47d38145b16ad8a788cc (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 23:30:16.390946 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 3565b68068904064a965cd0a82a20802 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 23:30:16.392004 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c7c13b00be47d38145b16ad8a788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.392400 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 07438388789747e68da9846e153a903b (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 23:30:16.369753 1781 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:16.398903 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c92025b82e6497ba49fb88677061e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.401252 1779 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:16.401778 1779 ts_tablet_manager.cc:1397] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.182s user 0.010s sys 0.003s
I20260927 23:30:16.402604 1786 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:16.403074 1786 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.403210 1724 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddb51781fec46b194174de5f4406def (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 23:30:16.404465 1724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddb51781fec46b194174de5f4406def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.413607 1778 tablet_bootstrap.cc:654] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.415256 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ed89195cf4ee49eeed018ec5039e7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 23:30:16.415844 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fafc0b4640efa0d437d468e2bdac (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 23:30:16.417070 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ed89195cf4ee49eeed018ec5039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.417508 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9629fafc0b4640efa0d437d468e2bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.436316 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565b68068904064a965cd0a82a20802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.442659 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0803b6723b37453ca7518fb7ced3081d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.457096 1779 raft_consensus.cc:348] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.457882 1779 raft_consensus.cc:374] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.458169 1779 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.467905 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07438388789747e68da9846e153a903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.460152 1786 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:16.481439 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 2686f0078ab94fbd99b4744ba43a0980 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 23:30:16.481957 1779 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.482766 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686f0078ab94fbd99b4744ba43a0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.484203 1779 ts_tablet_manager.cc:1428] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.082s user 0.007s sys 0.004s
I20260927 23:30:16.484480 1777 tablet_bootstrap.cc:654] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.485174 1779 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:16.485634 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 8c92025b82e6497ba49fb88677061e6e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260927 23:30:16.482520 1646 tablet_service.cc:1467] Processing CreateTablet for tablet f797da3c81264653acefad127c4e4b2e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 23:30:16.486860 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c92025b82e6497ba49fb88677061e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.501483 1574 tablet_service.cc:1467] Processing CreateTablet for tablet 58cf4a7fdcb146b0a364fa00e849e902 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 23:30:16.475443 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcc903f42c448e1beb437468961e2b0 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 23:30:16.504957 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e412df23ce8e4e71bc3c5bc171ca0e85 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 23:30:16.508656 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab559a361d74acca8ee9c195b91d830 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 23:30:16.514153 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 502de1f57afe4b808a200ece197c19e6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 23:30:16.523358 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797da3c81264653acefad127c4e4b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.531507 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 87786104826046e7852edc590e194f5d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 23:30:16.532059 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4c2a41313d4adcb6e037570a9d5882 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 23:30:16.532276 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 99c7c13b00be47d38145b16ad8a788cc (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 23:30:16.531555 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ed89195cf4ee49eeed018ec5039e7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 23:30:16.533335 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4c2a41313d4adcb6e037570a9d5882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.533739 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ed89195cf4ee49eeed018ec5039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.534061 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cf4a7fdcb146b0a364fa00e849e902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.536674 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 502de1f57afe4b808a200ece197c19e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.542930 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c7c13b00be47d38145b16ad8a788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.565135 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab559a361d74acca8ee9c195b91d830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.565161 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 7fcc903f42c448e1beb437468961e2b0 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 23:30:16.566413 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcc903f42c448e1beb437468961e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.566653 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fcc903f42c448e1beb437468961e2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.572739 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 9c719671f35640549c1b944112b3fc03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 23:30:16.574142 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c719671f35640549c1b944112b3fc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.575577 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddb51781fec46b194174de5f4406def (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 23:30:16.576932 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddb51781fec46b194174de5f4406def. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.579492 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87786104826046e7852edc590e194f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.559984 1779 tablet_bootstrap.cc:654] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.587735 1778 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:16.588213 1778 ts_tablet_manager.cc:1397] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.244s user 0.023s sys 0.012s
I20260927 23:30:16.597806 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 2686f0078ab94fbd99b4744ba43a0980 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 23:30:16.599159 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686f0078ab94fbd99b4744ba43a0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.611565 1785 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:16.612042 1785 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.606967 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e412df23ce8e4e71bc3c5bc171ca0e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.619000 1718 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4c2a41313d4adcb6e037570a9d5882 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 23:30:16.617221 1785 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:16.656932 1722 tablet_service.cc:1467] Processing CreateTablet for tablet f797da3c81264653acefad127c4e4b2e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 23:30:16.661790 1644 tablet_service.cc:1467] Processing CreateTablet for tablet fa9f6499247b4a1e9ef1440ac8906b50 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 23:30:16.663081 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9f6499247b4a1e9ef1440ac8906b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.672982 1777 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:16.650187 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4c2a41313d4adcb6e037570a9d5882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.673478 1777 ts_tablet_manager.cc:1397] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.355s user 0.026s sys 0.005s
I20260927 23:30:16.676427 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797da3c81264653acefad127c4e4b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.678447 1777 raft_consensus.cc:348] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.679190 1777 raft_consensus.cc:374] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.679448 1777 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.673830 1653 tablet_service.cc:1467] Processing CreateTablet for tablet a191cb6c79334070a9ff2a1a9db5604c (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 23:30:16.680752 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cb6c79334070a9ff2a1a9db5604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.683312 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:16.684098 1588 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:16.678205 1719 tablet_service.cc:1467] Processing CreateTablet for tablet e412df23ce8e4e71bc3c5bc171ca0e85 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 23:30:16.612042 1578 tablet_service.cc:1467] Processing CreateTablet for tablet 9629fafc0b4640efa0d437d468e2bdac (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 23:30:16.691278 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4ed89195cf4ee49eeed018ec5039e7 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 23:30:16.692479 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4ed89195cf4ee49eeed018ec5039e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.696026 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:16.696671 1663 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:16.698784 1551 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:16.699610 1786 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:16.698534 1777 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.701198 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab559a361d74acca8ee9c195b91d830 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 23:30:16.699962 1786 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:16.701625 1777 ts_tablet_manager.cc:1428] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260927 23:30:16.704725 1786 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:16.705596 1777 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:16.632668 1778 raft_consensus.cc:348] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.706818 1778 raft_consensus.cc:374] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.707217 1778 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.708187 1778 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.717020 1572 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4c2a41313d4adcb6e037570a9d5882 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 23:30:16.718156 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 2686f0078ab94fbd99b4744ba43a0980 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 23:30:16.718938 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e412df23ce8e4e71bc3c5bc171ca0e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.720124 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4c2a41313d4adcb6e037570a9d5882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.720592 1574 tablet_service.cc:1467] Processing CreateTablet for tablet f797da3c81264653acefad127c4e4b2e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 23:30:16.723409 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:16.726275 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 87786104826046e7852edc590e194f5d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 23:30:16.727995 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87786104826046e7852edc590e194f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.731966 1777 tablet_bootstrap.cc:654] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.733139 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 9c719671f35640549c1b944112b3fc03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 23:30:16.742452 1781 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:16.742936 1781 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.744797 1781 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:16.747157 1571 tablet_service.cc:1467] Processing CreateTablet for tablet 8c92025b82e6497ba49fb88677061e6e (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
W20260927 23:30:16.750068 1552 leader_election.cc:341] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.750806 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797da3c81264653acefad127c4e4b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.755362 1786 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.756295 1650 tablet_service.cc:1467] Processing CreateTablet for tablet e5355e82529343f5b479b2450cefa8ea (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260927 23:30:16.757570 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5355e82529343f5b479b2450cefa8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.758410 1786 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:16.758562 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:16.759397 1663 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:16.763043 1785 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:16.763515 1785 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.766386 1785 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:16.781834 1649 tablet_service.cc:1467] Processing CreateTablet for tablet f662be9387d148199907fd0e049ba4be (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 23:30:16.783085 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f662be9387d148199907fd0e049ba4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.788900 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:16.790910 1627 leader_election.cc:341] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.791728 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686f0078ab94fbd99b4744ba43a0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.803160 1652 tablet_service.cc:1467] Processing CreateTablet for tablet 861e779ad2d04547bd9be0bfbc2616c5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 23:30:16.804423 1652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e779ad2d04547bd9be0bfbc2616c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.807765 1779 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:16.808212 1779 ts_tablet_manager.cc:1397] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.323s user 0.027s sys 0.011s
I20260927 23:30:16.812242 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c719671f35640549c1b944112b3fc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.813928 1628 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:16.818550 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:16.820183 1627 leader_election.cc:341] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.822511 1779 raft_consensus.cc:348] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.823262 1779 raft_consensus.cc:374] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:16.823523 1779 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:16.794796 1651 tablet_service.cc:1467] Processing CreateTablet for tablet e6bcce456c08487491b881518f4f8b46 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 23:30:16.827152 1778 ts_tablet_manager.cc:1428] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.239s user 0.025s sys 0.039s
I20260927 23:30:16.828717 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bcce456c08487491b881518f4f8b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.828281 1779 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.814136 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 9154850d4ac4425dae5bbfde7004d809 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 23:30:16.833940 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:16.837106 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c92025b82e6497ba49fb88677061e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.837726 1778 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
W20260927 23:30:16.837986 1628 leader_election.cc:341] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.838462 1628 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:16.839874 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154850d4ac4425dae5bbfde7004d809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.843365 1576 tablet_service.cc:1467] Processing CreateTablet for tablet e412df23ce8e4e71bc3c5bc171ca0e85 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 23:30:16.845608 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 785f0fd9d6434fb98718143105ca55aa (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 23:30:16.846777 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785f0fd9d6434fb98718143105ca55aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.846447 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 08bafa5142a9452db744d03134c85be6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 23:30:16.847563 1779 ts_tablet_manager.cc:1428] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20260927 23:30:16.835696 1577 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46514 (request call id 30) took 1003 ms. Trace:
I20260927 23:30:16.847755 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bafa5142a9452db744d03134c85be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.849593 1646 tablet_service.cc:1467] Processing CreateTablet for tablet d0bbde2432f547cbab3588c16fb623cf (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 23:30:16.850216 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 18302bf8887a4eb685e0fa76572b5605 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 23:30:16.850986 1778 tablet_bootstrap.cc:654] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.851393 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18302bf8887a4eb685e0fa76572b5605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.855068 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bbde2432f547cbab3588c16fb623cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.864168 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:16.846447 1570 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab559a361d74acca8ee9c195b91d830 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 23:30:16.847939 1577 rpcz_store.cc:276] 0927 23:30:15.832026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:16.336990 (+504964us) service_pool.cc:227] Handling call
0927 23:30:16.391983 (+ 54993us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0927 23:30:16.392101 (+ 118us) ts_tablet_manager.cc:676] Creating new metadata...
0927 23:30:16.790349 (+398248us) tablet_metadata.cc:641] Metadata flushed
0927 23:30:16.835240 (+ 44891us) tablet_replica.cc:205] Creating consensus instance
0927 23:30:16.835642 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:30:16.869598 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:16.871287 1702 leader_election.cc:341] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:16.871908 1702 leader_election.cc:341] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.877404 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab559a361d74acca8ee9c195b91d830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.877200 1569 tablet_service.cc:1467] Processing CreateTablet for tablet 9c719671f35640549c1b944112b3fc03 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 23:30:16.878372 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c719671f35640549c1b944112b3fc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.878089 1575 tablet_service.cc:1467] Processing CreateTablet for tablet a191cb6c79334070a9ff2a1a9db5604c (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 23:30:16.879282 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cb6c79334070a9ff2a1a9db5604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.879876 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:16.881265 1552 leader_election.cc:341] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:16.882179 1720 tablet_service.cc:1467] Processing CreateTablet for tablet fa9f6499247b4a1e9ef1440ac8906b50 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 23:30:16.883347 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9629fafc0b4640efa0d437d468e2bdac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.886299 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab559a361d74acca8ee9c195b91d830. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.887122 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e412df23ce8e4e71bc3c5bc171ca0e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.875866 1663 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:16.890774 1781 raft_consensus.cc:2741] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 23:30:16.892426 1551 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:16.893648 1802 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:16.894142 1802 raft_consensus.cc:686] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:16.894935 1802 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:16.900331 1803 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:16.901755 1779 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:16.902519 1718 tablet_service.cc:1467] Processing CreateTablet for tablet e5355e82529343f5b479b2450cefa8ea (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260927 23:30:16.908062 1725 tablet_service.cc:1467] Processing CreateTablet for tablet e6bcce456c08487491b881518f4f8b46 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 23:30:16.909612 1574 tablet_service.cc:1467] Processing CreateTablet for tablet e5355e82529343f5b479b2450cefa8ea (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20260927 23:30:16.906505 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 68547.1us
I20260927 23:30:16.870007 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 87786104826046e7852edc590e194f5d (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 23:30:16.893677 1727 tablet_service.cc:1467] Processing CreateTablet for tablet a191cb6c79334070a9ff2a1a9db5604c (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 23:30:16.913341 1724 tablet_service.cc:1467] Processing CreateTablet for tablet f662be9387d148199907fd0e049ba4be (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 23:30:16.916875 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5355e82529343f5b479b2450cefa8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.919838 1722 tablet_service.cc:1467] Processing CreateTablet for tablet 785f0fd9d6434fb98718143105ca55aa (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 23:30:16.910836 1574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5355e82529343f5b479b2450cefa8ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.923398 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bcce456c08487491b881518f4f8b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.925158 1727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a191cb6c79334070a9ff2a1a9db5604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.921859 1719 tablet_service.cc:1467] Processing CreateTablet for tablet 08bafa5142a9452db744d03134c85be6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 23:30:16.927309 1571 tablet_service.cc:1467] Processing CreateTablet for tablet e6bcce456c08487491b881518f4f8b46 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 23:30:16.927711 1719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bafa5142a9452db744d03134c85be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.928596 1571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bcce456c08487491b881518f4f8b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.935250 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 861e779ad2d04547bd9be0bfbc2616c5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 23:30:16.936299 1722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785f0fd9d6434fb98718143105ca55aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.937050 1718 tablet_service.cc:1467] Processing CreateTablet for tablet d0bbde2432f547cbab3588c16fb623cf (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 23:30:16.941980 1718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bbde2432f547cbab3588c16fb623cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.941686 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 9154850d4ac4425dae5bbfde7004d809 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 23:30:16.944551 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9f6499247b4a1e9ef1440ac8906b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.950616 1724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f662be9387d148199907fd0e049ba4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.961431 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e779ad2d04547bd9be0bfbc2616c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.955113 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87786104826046e7852edc590e194f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:16.976226 1779 tablet_bootstrap.cc:654] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:16.977715 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:16.978338 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:16.979000 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154850d4ac4425dae5bbfde7004d809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.003228 1576 tablet_service.cc:1467] Processing CreateTablet for tablet 861e779ad2d04547bd9be0bfbc2616c5 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 23:30:17.006387 1576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e779ad2d04547bd9be0bfbc2616c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:30:17.006824 1700 leader_election.cc:341] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.007349 1700 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.008482 1785 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 23:30:17.018267 1700 leader_election.cc:341] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.018739 1700 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.019465 1785 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.021544 1570 tablet_service.cc:1467] Processing CreateTablet for tablet d0bbde2432f547cbab3588c16fb623cf (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 23:30:16.985443 1778 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.034175 1778 ts_tablet_manager.cc:1397] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.197s user 0.015s sys 0.030s
I20260927 23:30:17.034706 1575 tablet_service.cc:1467] Processing CreateTablet for tablet 18302bf8887a4eb685e0fa76572b5605 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 23:30:17.035912 1575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18302bf8887a4eb685e0fa76572b5605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.037065 1778 raft_consensus.cc:348] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.037865 1778 raft_consensus.cc:374] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.038239 1778 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.039084 1778 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.041497 1778 ts_tablet_manager.cc:1428] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:17.042425 1778 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:16.985167 1578 tablet_service.cc:1467] Processing CreateTablet for tablet 08bafa5142a9452db744d03134c85be6 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 23:30:16.979336 1484 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:16.988204 1720 tablet_service.cc:1467] Processing CreateTablet for tablet 18302bf8887a4eb685e0fa76572b5605 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 23:30:16.991104 1777 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:16.990738 1572 tablet_service.cc:1467] Processing CreateTablet for tablet fa9f6499247b4a1e9ef1440ac8906b50 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 23:30:17.003147 1573 tablet_service.cc:1467] Processing CreateTablet for tablet 785f0fd9d6434fb98718143105ca55aa (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 23:30:17.025499 1577 tablet_service.cc:1467] Processing CreateTablet for tablet 9154850d4ac4425dae5bbfde7004d809 (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 23:30:16.984715 1569 tablet_service.cc:1467] Processing CreateTablet for tablet f662be9387d148199907fd0e049ba4be (DEFAULT_TABLE table=test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 23:30:17.047187 1777 ts_tablet_manager.cc:1397] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.342s user 0.014s sys 0.000s
I20260927 23:30:17.049477 1779 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
W20260927 23:30:17.049603 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:17.050057 1777 raft_consensus.cc:348] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.050011 1779 ts_tablet_manager.cc:1397] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.009s
I20260927 23:30:17.050838 1777 raft_consensus.cc:374] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.051146 1777 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.051681 1777 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.053503 1777 ts_tablet_manager.cc:1428] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 23:30:17.047541 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18302bf8887a4eb685e0fa76572b5605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.053762 1779 raft_consensus.cc:348] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.055130 1779 raft_consensus.cc:374] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.055471 1779 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.056115 1779 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.058202 1779 ts_tablet_manager.cc:1428] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260927 23:30:17.059033 1572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9f6499247b4a1e9ef1440ac8906b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.059643 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f662be9387d148199907fd0e049ba4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.060479 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785f0fd9d6434fb98718143105ca55aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.061278 1577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154850d4ac4425dae5bbfde7004d809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.061849 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08bafa5142a9452db744d03134c85be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.064554 1570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bbde2432f547cbab3588c16fb623cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:17.065256 1781 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.065861 1777 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.065805 1781 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.069634 1779 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.086124 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:17.087577 1628 leader_election.cc:341] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.095929 1779 tablet_bootstrap.cc:654] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.103708 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:17.105201 1627 leader_election.cc:341] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.105608 1627 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.106002 1781 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.106516 1803 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.111637 1778 tablet_bootstrap.cc:654] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.135046 1779 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.135504 1779 ts_tablet_manager.cc:1397] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.023s
I20260927 23:30:17.138957 1778 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.139561 1778 ts_tablet_manager.cc:1397] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.006s
I20260927 23:30:17.141345 1777 tablet_bootstrap.cc:654] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.142202 1778 raft_consensus.cc:348] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.143085 1778 raft_consensus.cc:374] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.142897 1779 raft_consensus.cc:348] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.143465 1778 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.143735 1779 raft_consensus.cc:374] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.144054 1779 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.144349 1778 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.144821 1779 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.146735 1778 ts_tablet_manager.cc:1428] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:30:17.146970 1779 ts_tablet_manager.cc:1428] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260927 23:30:17.147715 1778 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.147925 1779 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.153795 1779 tablet_bootstrap.cc:654] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.155428 1778 tablet_bootstrap.cc:654] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.160017 1777 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.160705 1777 ts_tablet_manager.cc:1397] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.017s
I20260927 23:30:17.164378 1777 raft_consensus.cc:348] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.165452 1777 raft_consensus.cc:374] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.165868 1777 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.166802 1777 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.169287 1777 ts_tablet_manager.cc:1428] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:30:17.170516 1777 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.174526 1779 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.174925 1779 ts_tablet_manager.cc:1397] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.008s
I20260927 23:30:17.176578 1777 tablet_bootstrap.cc:654] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.177122 1779 raft_consensus.cc:348] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.177661 1779 raft_consensus.cc:374] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.177891 1779 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.178480 1779 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.183889 1779 ts_tablet_manager.cc:1428] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 23:30:17.184854 1779 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.187930 1777 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.188444 1777 ts_tablet_manager.cc:1397] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20260927 23:30:17.189644 1779 tablet_bootstrap.cc:654] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.190222 1778 tablet_bootstrap.cc:492] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.190860 1778 ts_tablet_manager.cc:1397] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.006s
I20260927 23:30:17.191092 1777 raft_consensus.cc:348] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.192306 1777 raft_consensus.cc:374] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.192584 1777 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.193787 1802 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.194268 1802 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.193737 1777 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.194499 1778 raft_consensus.cc:348] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.195509 1778 raft_consensus.cc:374] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.195863 1778 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.196247 1802 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.196689 1778 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.197834 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.199673 1778 ts_tablet_manager.cc:1428] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260927 23:30:17.199930 1777 ts_tablet_manager.cc:1428] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260927 23:30:17.199783 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.200692 1779 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.200834 1777 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.201121 1779 ts_tablet_manager.cc:1397] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
W20260927 23:30:17.201195 1552 leader_election.cc:341] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:17.202220 1551 leader_election.cc:341] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.202663 1551 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.203665 1778 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.203888 1786 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.203821 1779 raft_consensus.cc:348] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.204594 1779 raft_consensus.cc:374] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.204909 1779 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.205770 1779 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.210218 1778 tablet_bootstrap.cc:654] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:17.212430 1614 tablet.cc:2369] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:30:17.214912 1779 ts_tablet_manager.cc:1428] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260927 23:30:17.215595 1778 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.215772 1779 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.216035 1778 ts_tablet_manager.cc:1397] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260927 23:30:17.218833 1778 raft_consensus.cc:348] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.219520 1778 raft_consensus.cc:374] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.219807 1778 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.221022 1779 tablet_bootstrap.cc:654] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.220674 1778 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.223448 1777 tablet_bootstrap.cc:654] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.227785 1785 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.228395 1785 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.229302 1778 ts_tablet_manager.cc:1428] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260927 23:30:17.230583 1778 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.232386 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.233283 1588 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.234936 1777 tablet_bootstrap.cc:492] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.234777 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.235420 1777 ts_tablet_manager.cc:1397] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.012s
I20260927 23:30:17.236790 1778 tablet_bootstrap.cc:654] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.238044 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.238750 1785 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.239012 1662 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.238260 1777 raft_consensus.cc:348] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.240362 1777 raft_consensus.cc:374] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.240648 1777 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.241254 1785 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.241703 1785 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.241317 1777 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.242015 1785 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.242651 1779 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.243083 1779 ts_tablet_manager.cc:1397] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.021s
I20260927 23:30:17.245813 1777 ts_tablet_manager.cc:1428] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.010s user 0.001s sys 0.007s
I20260927 23:30:17.245674 1779 raft_consensus.cc:348] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.246402 1779 raft_consensus.cc:374] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.246656 1779 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.247337 1779 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.249279 1779 ts_tablet_manager.cc:1428] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 23:30:17.250099 1779 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.256711 1778 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.256800 1779 tablet_bootstrap.cc:654] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.257283 1778 ts_tablet_manager.cc:1397] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.009s
I20260927 23:30:17.259531 1803 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.259925 1778 raft_consensus.cc:348] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.249325 1785 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.260685 1778 raft_consensus.cc:374] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.260108 1803 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.260996 1778 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.261751 1778 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.262907 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.263461 1588 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.264256 1778 ts_tablet_manager.cc:1428] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 23:30:17.265188 1778 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.266785 1803 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.267473 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.269148 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:17.270495 1627 leader_election.cc:341] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.270958 1778 tablet_bootstrap.cc:654] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:17.271536 1628 leader_election.cc:341] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.271996 1802 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.271947 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.272472 1802 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.274283 1802 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.275923 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:17.277552 1551 leader_election.cc:341] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.277740 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:17.279778 1552 leader_election.cc:341] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.280336 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.281276 1802 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.256945 1777 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.286500 1803 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.287067 1777 tablet_bootstrap.cc:654] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.289386 1785 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.290958 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.291603 1662 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.293054 1778 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.293612 1778 ts_tablet_manager.cc:1397] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.017s
I20260927 23:30:17.294453 1588 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.295581 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.296423 1785 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.296875 1785 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:17.296964 1778 raft_consensus.cc:348] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.297649 1778 raft_consensus.cc:374] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.298272 1778 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.297654 1785 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.299149 1778 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.301059 1778 ts_tablet_manager.cc:1428] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 23:30:17.301707 1662 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.302169 1778 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.304118 1779 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.304533 1779 ts_tablet_manager.cc:1397] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.023s
I20260927 23:30:17.306924 1779 raft_consensus.cc:348] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.307657 1779 raft_consensus.cc:374] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.307786 1778 tablet_bootstrap.cc:654] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.307958 1779 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.308636 1779 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.310611 1779 ts_tablet_manager.cc:1428] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 23:30:17.311599 1779 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.317902 1777 tablet_bootstrap.cc:492] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.318508 1779 tablet_bootstrap.cc:654] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.318437 1777 ts_tablet_manager.cc:1397] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.004s
I20260927 23:30:17.321858 1777 raft_consensus.cc:348] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.322366 1778 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.322588 1777 raft_consensus.cc:374] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.322890 1778 ts_tablet_manager.cc:1397] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260927 23:30:17.325771 1778 raft_consensus.cc:348] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.322868 1777 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.327361 1777 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.329694 1777 ts_tablet_manager.cc:1428] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 23:30:17.330487 1778 raft_consensus.cc:374] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.330862 1778 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.331763 1778 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.332820 1777 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.334139 1778 ts_tablet_manager.cc:1428] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 23:30:17.333066 1483 catalog_manager.cc:5495] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:17.338063 1777 tablet_bootstrap.cc:654] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.340454 1779 tablet_bootstrap.cc:492] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.340862 1779 ts_tablet_manager.cc:1397] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.000s
I20260927 23:30:17.341104 1785 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.341530 1785 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.343116 1779 raft_consensus.cc:348] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.343832 1779 raft_consensus.cc:374] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.343915 1785 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.344096 1779 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.346585 1778 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.346253 1779 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.348104 1779 ts_tablet_manager.cc:1428] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:30:17.348435 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.348894 1779 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.350569 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.351298 1662 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:17.352201 1778 tablet_bootstrap.cc:654] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.353616 1700 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.354699 1785 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.355219 1779 tablet_bootstrap.cc:654] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.361498 1785 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.365777 1777 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.366232 1777 ts_tablet_manager.cc:1397] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20260927 23:30:17.366518 1778 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.367202 1778 ts_tablet_manager.cc:1397] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260927 23:30:17.368544 1777 raft_consensus.cc:348] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.369215 1777 raft_consensus.cc:374] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.369454 1777 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.369450 1802 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:17.369663 1778 raft_consensus.cc:348] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.370625 1778 raft_consensus.cc:374] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.370410 1777 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.371491 1802 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.371980 1802 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.372824 1777 ts_tablet_manager.cc:1428] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 23:30:17.374151 1777 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.375382 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:17.377077 1551 leader_election.cc:341] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.378438 1822 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.379050 1822 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.380895 1777 tablet_bootstrap.cc:654] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.383253 1822 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.384857 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.385794 1588 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.387655 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.388484 1662 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.390007 1700 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.392253 1822 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.392701 1822 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.393151 1822 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.400628 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.401547 1736 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:17.402158 1777 tablet_bootstrap.cc:492] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.402902 1777 ts_tablet_manager.cc:1397] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.008s
I20260927 23:30:17.403486 1552 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.404544 1786 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.404980 1786 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.405452 1786 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.407095 1777 raft_consensus.cc:348] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.408105 1777 raft_consensus.cc:374] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.408528 1777 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.409574 1777 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.410634 1802 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.421391 1786 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.424842 1779 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.426549 1822 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.427373 1779 ts_tablet_manager.cc:1397] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.079s user 0.031s sys 0.039s
I20260927 23:30:17.429152 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.425625 1777 ts_tablet_manager.cc:1428] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260927 23:30:17.429546 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.429813 1662 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.430102 1588 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.431916 1777 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.370999 1778 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.436316 1588 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.436867 1778 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.437407 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.438180 1785 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.439904 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.440237 1778 ts_tablet_manager.cc:1428] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.072s user 0.005s sys 0.000s
I20260927 23:30:17.440518 1663 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.442705 1786 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.443936 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:17.444530 1736 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.446205 1777 tablet_bootstrap.cc:654] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.447775 1778 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.434919 1779 raft_consensus.cc:348] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.448227 1779 raft_consensus.cc:374] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.448622 1779 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.449838 1779 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.456108 1736 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.457156 1552 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.457978 1786 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.458613 1786 raft_consensus.cc:686] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:17.459591 1786 consensus_queue.cc:237] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.463203 1663 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.436321 1662 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.472813 1785 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:17.475327 1785 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.487779 1778 tablet_bootstrap.cc:654] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.488754 1822 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.490760 1838 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.491408 1838 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.494506 1778 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.494936 1778 ts_tablet_manager.cc:1397] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.001s
I20260927 23:30:17.495604 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:17.497232 1551 leader_election.cc:341] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.497786 1778 raft_consensus.cc:348] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.498483 1778 raft_consensus.cc:374] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.500826 1778 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.501627 1777 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.501482 1778 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.502244 1777 ts_tablet_manager.cc:1397] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.071s user 0.021s sys 0.016s
I20260927 23:30:17.501998 1483 catalog_manager.cc:5495] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:17.503459 1778 ts_tablet_manager.cc:1428] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 23:30:17.504254 1778 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.504669 1786 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:17.505579 1779 ts_tablet_manager.cc:1428] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.078s user 0.004s sys 0.028s
I20260927 23:30:17.506294 1836 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.506832 1779 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.506796 1836 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.510296 1778 tablet_bootstrap.cc:654] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.510469 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.509429 1777 raft_consensus.cc:348] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
W20260927 23:30:17.511927 1702 leader_election.cc:341] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.512104 1777 raft_consensus.cc:374] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.512537 1777 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.513670 1803 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.514303 1803 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.518364 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.519416 1836 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:17.520169 1628 leader_election.cc:341] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.521685 1803 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.522428 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.523118 1777 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
W20260927 23:30:17.523944 1627 leader_election.cc:341] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.522533 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.525734 1627 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.526060 1802 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.529268 1777 ts_tablet_manager.cc:1428] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260927 23:30:17.528723 1802 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.529801 1803 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.530278 1777 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.530452 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.530977 1802 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.531128 1736 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:17.532655 1552 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.536235 1778 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.536659 1778 ts_tablet_manager.cc:1397] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.010s
W20260927 23:30:17.538461 1700 leader_election.cc:341] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.538674 1803 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.538916 1700 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.539201 1803 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.540215 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.540845 1736 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:17.541326 1803 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.541534 1836 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.542827 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.545240 1779 tablet_bootstrap.cc:654] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.544354 1482 catalog_manager.cc:5495] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:17.533247 1838 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.549471 1778 raft_consensus.cc:348] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.550506 1778 raft_consensus.cc:374] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.550894 1778 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.546468 1802 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.551712 1778 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.553113 1802 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.554574 1802 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.555648 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:17.556969 1628 leader_election.cc:341] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.558880 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.560961 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.559561 1663 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:17.547837 1786 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260927 23:30:17.563153 1627 leader_election.cc:341] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.563565 1627 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.564265 1781 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.562695 1786 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.567072 1778 ts_tablet_manager.cc:1428] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260927 23:30:17.568015 1778 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.569314 1803 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.569851 1803 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.574538 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.575376 1588 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:17.576833 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:17.567162 1786 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.577957 1781 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.578471 1781 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.578868 1781 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.581949 1838 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.582472 1838 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.583282 1802 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.585529 1802 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.586772 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.587307 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.587231 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.587875 1803 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.587991 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:17.588570 1736 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.588606 1737 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:17.588979 1781 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.589640 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.591464 1781 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.592132 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.592660 1588 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.588171 1838 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:17.596475 1551 leader_election.cc:341] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:17.598122 1551 leader_election.cc:341] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.596210 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:17.599715 1627 leader_election.cc:341] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.602385 1736 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.602713 1588 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:17.603912 1552 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.605512 1838 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.605960 1777 tablet_bootstrap.cc:654] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.593969 1802 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.606557 1838 raft_consensus.cc:686] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:17.607997 1802 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.608214 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.608804 1802 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.609346 1803 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.608970 1838 consensus_queue.cc:237] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.600425 1781 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.610193 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.621417 1803 raft_consensus.cc:686] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:17.623708 1803 consensus_queue.cc:237] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:17.630101 1627 leader_election.cc:341] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.620584 1781 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.631273 1785 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.632406 1778 tablet_bootstrap.cc:654] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.632133 1785 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.632989 1786 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.636644 1785 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.638100 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.638871 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.639690 1661 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.640481 1588 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.640642 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.640995 1802 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.641567 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:17.641650 1587 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:17.641956 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.643062 1802 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.644189 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.644392 1786 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.644686 1628 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:17.644043 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:17.645890 1736 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:17.646418 1551 leader_election.cc:341] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.647198 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.647716 1661 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.642167 1737 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.656993 1661 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.652112 1779 tablet_bootstrap.cc:492] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.658615 1779 ts_tablet_manager.cc:1397] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.152s user 0.017s sys 0.013s
I20260927 23:30:17.654489 1785 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.658141 1551 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:17.663643 1786 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.664471 1786 raft_consensus.cc:686] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:17.664922 1779 raft_consensus.cc:348] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.664567 1781 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.665686 1779 raft_consensus.cc:374] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.666033 1781 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.666448 1836 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.666610 1781 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.667114 1781 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.666917 1836 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.668951 1836 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.671070 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.672667 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.673403 1587 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.666004 1779 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.676129 1702 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.662300 1785 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.678143 1785 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.678355 1836 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.678690 1836 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.678988 1836 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.683827 1781 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.693375 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:17.695559 1822 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.695780 1835 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:17.696137 1822 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.698532 1822 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.722468 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.723348 1588 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.724908 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.726004 1822 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.726451 1822 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.726861 1822 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.698086 1781 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.767576 1779 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.771612 1836 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.774389 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:17.711306 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59871.5us
I20260927 23:30:17.713510 1736 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.776464 1779 ts_tablet_manager.cc:1428] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.118s user 0.004s sys 0.003s
I20260927 23:30:17.777878 1779 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.665688 1786 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } }
I20260927 23:30:17.778796 1588 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.774977 1836 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.786000 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.787289 1838 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.787985 1838 raft_consensus.cc:686] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:17.788746 1838 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.790195 1588 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.791661 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.791643 1702 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.792375 1588 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.792922 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.791647 1483 catalog_manager.cc:5495] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:17.794164 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.795091 1661 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.796029 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.799261 1588 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:17.800983 1822 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.802196 1836 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.802733 1836 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:17.803525 1836 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:17.804243 1700 leader_election.cc:341] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.804733 1628 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.805058 1785 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:17.805641 1700 leader_election.cc:341] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.806493 1735 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 1: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:17.810124 1785 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.810765 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.811396 1588 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:17.813279 1737 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:17.815874 1777 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.816335 1777 ts_tablet_manager.cc:1397] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.286s user 0.021s sys 0.024s
I20260927 23:30:17.823994 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.825029 1663 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:17.826454 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:17.828929 1777 raft_consensus.cc:348] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.829638 1777 raft_consensus.cc:374] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.829892 1777 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.830652 1777 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.832075 1853 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.832667 1853 raft_consensus.cc:686] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:17.833676 1853 consensus_queue.cc:237] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.840205 1777 ts_tablet_manager.cc:1428] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20260927 23:30:17.841315 1777 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.842213 1835 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.843076 1835 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.845966 1835 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.850353 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.850996 1835 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.851179 1588 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.854978 1778 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:17.855451 1778 ts_tablet_manager.cc:1397] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.288s user 0.038s sys 0.069s
I20260927 23:30:17.857383 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:17.858016 1586 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.859575 1779 tablet_bootstrap.cc:654] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.859484 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.860139 1663 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:17.861243 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.854130 1822 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:17.861781 1661 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.864099 1588 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.865357 1700 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.867965 1822 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:17.876950 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.877261 1822 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:17.876543 1777 tablet_bootstrap.cc:654] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.878955 1822 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.873945 1485 catalog_manager.cc:5495] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:17.883417 1835 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:17.879134 1836 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.885905 1661 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.882067 1778 raft_consensus.cc:348] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.886921 1778 raft_consensus.cc:374] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.887193 1778 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.886061 1836 raft_consensus.cc:686] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:17.888345 1836 consensus_queue.cc:237] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.888985 1778 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.890679 1786 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:17.891121 1786 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.893215 1786 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:17.893653 1778 ts_tablet_manager.cc:1428] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260927 23:30:17.895951 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:17.897918 1483 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:17.900273 1778 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:17.900094 1835 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:17.901434 1486 catalog_manager.cc:5495] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:17.905267 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:17.936981 1822 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:17.931090 1551 leader_election.cc:341] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.944012 1779 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:17.944514 1779 ts_tablet_manager.cc:1397] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.167s user 0.014s sys 0.010s
I20260927 23:30:17.944924 1822 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:17.945201 1552 leader_election.cc:341] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:17.946674 1778 tablet_bootstrap.cc:654] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.945629 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:17.949167 1777 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:17.949580 1777 ts_tablet_manager.cc:1397] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.015s
I20260927 23:30:17.944505 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:17.949510 1838 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:17.950347 1588 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.952121 1777 raft_consensus.cc:348] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.953752 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:17.954303 1663 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:17.956180 1838 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:17.952808 1777 raft_consensus.cc:374] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.958477 1777 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.960529 1777 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.963796 1663 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.964855 1700 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:17.965837 1822 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:17.966308 1822 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:17.960649 1779 raft_consensus.cc:348] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.967532 1777 ts_tablet_manager.cc:1428] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.018s user 0.002s sys 0.005s
I20260927 23:30:17.969206 1779 raft_consensus.cc:374] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:17.967491 1822 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.969877 1777 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:17.969779 1779 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:17.970664 1779 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:17.972853 1779 ts_tablet_manager.cc:1428] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260927 23:30:17.973835 1779 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:17.982156 1777 tablet_bootstrap.cc:654] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.995404 1588 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:17.995896 1779 tablet_bootstrap.cc:654] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:17.999662 1786 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.014266 1482 catalog_manager.cc:5495] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:18.017462 1482 catalog_manager.cc:5495] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } }
W20260927 23:30:18.029126 1551 consensus_peers.cc:487] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Couldn't send request to peer 1b945905616b4989853b0a92bfbd6b98. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260927 23:30:18.052866 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:18.056833 1778 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.057279 1778 ts_tablet_manager.cc:1397] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.157s user 0.018s sys 0.003s
I20260927 23:30:18.060317 1778 raft_consensus.cc:348] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.060971 1778 raft_consensus.cc:374] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.061239 1778 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.063249 1803 consensus_queue.cc:1035] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:30:18.067641 1627 consensus_peers.cc:487] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 -> Peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Couldn't send request to peer a2a3317e8da742d9aefe4c251abd4566. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:18.068415 1822 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.061830 1778 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.075418 1778 ts_tablet_manager.cc:1428] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260927 23:30:18.076342 1778 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.077132 1777 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.077521 1777 ts_tablet_manager.cc:1397] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.008s
I20260927 23:30:18.081117 1486 catalog_manager.cc:5495] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:18.083290 1486 catalog_manager.cc:5495] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:18.088356 1777 raft_consensus.cc:348] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.089043 1777 raft_consensus.cc:374] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.089314 1777 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.089635 1853 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.090093 1853 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.089977 1777 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.092458 1779 tablet_bootstrap.cc:492] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.092674 1777 ts_tablet_manager.cc:1428] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20260927 23:30:18.092890 1779 ts_tablet_manager.cc:1397] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.119s user 0.026s sys 0.042s
I20260927 23:30:18.093000 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.093494 1777 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.093834 1737 raft_consensus.cc:2379] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:18.095167 1779 raft_consensus.cc:348] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.095871 1779 raft_consensus.cc:374] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.096149 1779 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.096706 1778 tablet_bootstrap.cc:654] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.097276 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.098906 1628 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.099731 1777 tablet_bootstrap.cc:654] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.100304 1854 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.099665 1853 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.096791 1779 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.108661 1854 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.111943 1777 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.112408 1777 ts_tablet_manager.cc:1397] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260927 23:30:18.115677 1777 raft_consensus.cc:348] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.116369 1777 raft_consensus.cc:374] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.116636 1777 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.117415 1777 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.119963 1777 ts_tablet_manager.cc:1428] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.007s user 0.004s sys 0.005s
I20260927 23:30:18.120934 1777 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.122804 1778 tablet_bootstrap.cc:492] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.123292 1778 ts_tablet_manager.cc:1397] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.006s
I20260927 23:30:18.125643 1778 raft_consensus.cc:348] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.126430 1778 raft_consensus.cc:374] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.126742 1778 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.126911 1777 tablet_bootstrap.cc:654] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.127733 1778 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.130229 1778 ts_tablet_manager.cc:1428] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260927 23:30:18.131210 1778 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.132858 1786 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.133301 1786 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.135245 1786 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.137162 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.137817 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.138293 1737 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.139490 1552 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:18.140334 1786 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.140713 1786 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.140849 1777 tablet_bootstrap.cc:492] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.141180 1786 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.141400 1777 ts_tablet_manager.cc:1397] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260927 23:30:18.138692 1663 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.144846 1777 raft_consensus.cc:348] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.145565 1777 raft_consensus.cc:374] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.110399 1779 ts_tablet_manager.cc:1428] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20260927 23:30:18.145836 1777 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.146562 1777 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.147564 1778 tablet_bootstrap.cc:654] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.149168 1777 ts_tablet_manager.cc:1428] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260927 23:30:18.150069 1777 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.150665 1786 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.152376 1779 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.155654 1777 tablet_bootstrap.cc:654] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.157140 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:18.157773 1663 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.165056 1786 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.159824 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.170698 1737 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.166653 1777 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.174685 1777 ts_tablet_manager.cc:1397] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20260927 23:30:18.174994 1663 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.176694 1737 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.158181 1779 tablet_bootstrap.cc:654] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.176982 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.177057 1777 raft_consensus.cc:348] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.177575 1778 tablet_bootstrap.cc:492] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.177863 1777 raft_consensus.cc:374] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.178298 1786 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.178807 1778 ts_tablet_manager.cc:1397] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.019s
I20260927 23:30:18.178752 1777 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.179977 1786 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:18.179692 1777 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.180931 1786 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.181473 1778 raft_consensus.cc:348] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.182093 1777 ts_tablet_manager.cc:1428] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260927 23:30:18.182126 1778 raft_consensus.cc:374] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.182667 1778 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.183379 1777 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.183671 1778 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.186261 1778 ts_tablet_manager.cc:1428] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:18.187125 1778 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.191826 1777 tablet_bootstrap.cc:654] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.198594 1822 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.199266 1822 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.200656 1778 tablet_bootstrap.cc:654] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.203025 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.206938 1854 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.207619 1854 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.211171 1778 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.211340 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.211616 1778 ts_tablet_manager.cc:1397] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.000s
I20260927 23:30:18.211470 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.214118 1853 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.214816 1778 raft_consensus.cc:348] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.214743 1853 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.215574 1778 raft_consensus.cc:374] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.215873 1778 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.215708 1482 catalog_manager.cc:5495] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:18.216462 1778 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.217273 1853 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.220481 1700 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:18.221571 1785 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 23:30:18.222713 1627 leader_election.cc:341] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.223842 1778 ts_tablet_manager.cc:1428] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260927 23:30:18.223771 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.224995 1778 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.226688 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.228645 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:18.230427 1628 leader_election.cc:341] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.224750 1587 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:18.231989 1778 tablet_bootstrap.cc:654] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.233070 1822 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.231154 1854 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.232800 1628 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:18.235332 1627 leader_election.cc:341] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.235973 1854 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.235940 1627 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.236764 1854 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.237213 1854 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.237542 1853 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.240577 1779 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.241133 1779 ts_tablet_manager.cc:1397] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.089s user 0.018s sys 0.009s
I20260927 23:30:18.243021 1854 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.244182 1779 raft_consensus.cc:348] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.246838 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:18.247045 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.245546 1854 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.252671 1853 consensus_queue.cc:1035] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260927 23:30:18.251909 1822 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:18.263813 1627 leader_election.cc:341] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.263489 1587 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.264129 1822 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.264504 1777 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.265182 1777 ts_tablet_manager.cc:1397] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.062s
I20260927 23:30:18.266841 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.267480 1588 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:18.268232 1777 raft_consensus.cc:348] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.269111 1777 raft_consensus.cc:374] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.269001 1702 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:18.269476 1777 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.269889 1785 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.270249 1785 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.270670 1785 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.270288 1777 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.273123 1777 ts_tablet_manager.cc:1428] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 23:30:18.279629 1786 consensus_queue.cc:1035] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.285889 1836 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.288164 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.288326 1786 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.288822 1663 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:18.292053 1777 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.299527 1587 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:18.306519 1835 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.307894 1777 tablet_bootstrap.cc:654] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.310273 1663 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.315073 1628 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.316167 1853 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.316608 1853 raft_consensus.cc:686] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:18.317467 1853 consensus_queue.cc:237] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.284768 1822 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.299453 1785 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.323836 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:18.324440 1662 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.323836 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:18.326948 1587 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.245062 1779 raft_consensus.cc:374] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.328536 1779 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.334777 1779 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.336879 1779 ts_tablet_manager.cc:1428] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.095s user 0.006s sys 0.002s
I20260927 23:30:18.340546 1822 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.347718 1786 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.353260 1779 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.360821 1587 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:18.361966 1702 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:18.362799 1822 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.363246 1822 raft_consensus.cc:686] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:18.336563 1662 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:18.354990 1785 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.374003 1854 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.374745 1854 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.377058 1854 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.369805 1822 consensus_queue.cc:237] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.380111 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:18.385181 1627 leader_election.cc:341] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.382592 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:18.387583 1628 leader_election.cc:341] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.388042 1628 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.389101 1802 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.389683 1781 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.393663 1854 consensus_queue.cc:1035] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 23:30:18.399173 1802 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.423543 1835 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.425172 1778 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.425602 1778 ts_tablet_manager.cc:1397] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.201s user 0.057s sys 0.095s
I20260927 23:30:18.433020 1778 raft_consensus.cc:348] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.433604 1778 raft_consensus.cc:374] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.433832 1778 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.434441 1778 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.436754 1778 ts_tablet_manager.cc:1428] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260927 23:30:18.437054 1779 tablet_bootstrap.cc:654] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.438983 1778 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.444545 1777 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.445271 1777 ts_tablet_manager.cc:1397] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.153s user 0.030s sys 0.030s
I20260927 23:30:18.472875 1835 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.474038 1835 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.478087 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:18.479516 1702 leader_election.cc:341] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.490238 1835 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.490784 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.491168 1786 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.491701 1786 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:18.492345 1700 leader_election.cc:341] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.492894 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:18.493750 1786 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.494020 1835 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.494982 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.495537 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:18.496184 1552 leader_election.cc:341] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.497059 1835 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.512219 1483 catalog_manager.cc:5495] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:18.519744 1778 tablet_bootstrap.cc:654] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.520422 1777 raft_consensus.cc:348] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.521293 1777 raft_consensus.cc:374] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.521709 1777 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.521018 1822 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 23:30:18.523356 1777 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.525372 1661 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.528551 1551 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.535954 1802 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.536456 1802 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.536942 1802 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.542613 1779 tablet_bootstrap.cc:492] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.543128 1779 ts_tablet_manager.cc:1397] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.190s user 0.012s sys 0.018s
I20260927 23:30:18.546437 1779 raft_consensus.cc:348] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.547243 1779 raft_consensus.cc:374] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.547675 1779 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.547274 1777 ts_tablet_manager.cc:1428] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.102s user 0.005s sys 0.004s
I20260927 23:30:18.549297 1779 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.550585 1777 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.556924 1779 ts_tablet_manager.cc:1428] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260927 23:30:18.557768 1779 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.564379 1785 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.566450 1734 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.564921 1785 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.573395 1785 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.580372 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.581355 1802 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.588884 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.601315 1486 catalog_manager.cc:5495] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:18.608103 1586 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:18.614058 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:18.614575 1848 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.615216 1785 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.615046 1848 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.616797 1848 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.619395 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:18.620311 1660 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.622671 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.622901 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.626287 1627 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.630102 1781 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.631021 1661 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.626880 1802 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.632781 1778 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.633205 1778 ts_tablet_manager.cc:1397] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.194s user 0.006s sys 0.019s
I20260927 23:30:18.633311 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.636603 1777 tablet_bootstrap.cc:654] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:18.636622 1552 leader_election.cc:341] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.639786 1785 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.640091 1779 tablet_bootstrap.cc:654] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.647121 1660 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.658991 1551 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.659853 1802 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.660328 1802 raft_consensus.cc:686] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:18.661120 1802 consensus_queue.cc:237] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.661639 1779 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.662349 1779 ts_tablet_manager.cc:1397] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.004s
I20260927 23:30:18.662894 1778 raft_consensus.cc:348] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.663525 1778 raft_consensus.cc:374] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.663767 1778 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.664306 1778 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.669263 1778 ts_tablet_manager.cc:1428] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260927 23:30:18.671728 1779 raft_consensus.cc:348] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.672844 1779 raft_consensus.cc:374] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.673170 1779 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.673903 1779 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.676306 1838 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.677505 1779 ts_tablet_manager.cc:1428] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260927 23:30:18.679005 1778 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.677702 1777 tablet_bootstrap.cc:492] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.679919 1777 ts_tablet_manager.cc:1397] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.017s
I20260927 23:30:18.680763 1779 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.681845 1777 raft_consensus.cc:348] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.682839 1777 raft_consensus.cc:374] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.683084 1777 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.683734 1777 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.686161 1777 ts_tablet_manager.cc:1428] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 23:30:18.686992 1777 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.687783 1778 tablet_bootstrap.cc:654] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.692312 1779 tablet_bootstrap.cc:654] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.692858 1777 tablet_bootstrap.cc:654] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.705320 1779 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.704706 1906 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:18.705778 1779 ts_tablet_manager.cc:1397] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.008s
I20260927 23:30:18.708151 1779 raft_consensus.cc:348] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.708822 1779 raft_consensus.cc:374] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.709082 1779 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.709681 1779 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.713330 1779 ts_tablet_manager.cc:1428] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260927 23:30:18.714206 1779 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.714239 1778 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.714879 1778 ts_tablet_manager.cc:1397] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.017s
I20260927 23:30:18.717625 1777 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.718055 1777 ts_tablet_manager.cc:1397] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.005s
I20260927 23:30:18.717316 1778 raft_consensus.cc:348] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.719964 1778 raft_consensus.cc:374] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.722739 1778 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.723538 1778 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.724186 1777 raft_consensus.cc:348] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.725118 1777 raft_consensus.cc:374] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.725368 1777 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.725852 1778 ts_tablet_manager.cc:1428] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260927 23:30:18.726038 1777 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.726881 1778 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.727823 1779 tablet_bootstrap.cc:654] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.728130 1777 ts_tablet_manager.cc:1428] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260927 23:30:18.728935 1777 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.732875 1778 tablet_bootstrap.cc:654] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.734302 1777 tablet_bootstrap.cc:654] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.740576 1779 tablet_bootstrap.cc:492] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.740994 1779 ts_tablet_manager.cc:1397] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.008s
I20260927 23:30:18.743556 1779 raft_consensus.cc:348] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.744207 1779 raft_consensus.cc:374] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.744467 1779 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.745041 1779 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.746781 1779 ts_tablet_manager.cc:1428] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 23:30:18.747614 1779 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.747979 1485 catalog_manager.cc:5495] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:18.762596 1779 tablet_bootstrap.cc:654] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.763409 1777 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.763842 1777 ts_tablet_manager.cc:1397] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.004s
I20260927 23:30:18.766547 1778 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.766500 1777 raft_consensus.cc:348] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.767004 1778 ts_tablet_manager.cc:1397] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.002s
I20260927 23:30:18.767277 1777 raft_consensus.cc:374] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.767554 1777 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.768170 1777 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.769440 1778 raft_consensus.cc:348] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.770079 1778 raft_consensus.cc:374] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.770156 1777 ts_tablet_manager.cc:1428] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 23:30:18.770368 1778 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.771140 1778 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.772437 1777 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.774135 1778 ts_tablet_manager.cc:1428] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:30:18.775336 1778 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.776495 1779 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.776487 1848 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.777032 1848 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.778667 1777 tablet_bootstrap.cc:654] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.778981 1848 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.780732 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.781222 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:18.782027 1628 leader_election.cc:341] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:18.782614 1627 leader_election.cc:341] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.782991 1627 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.783967 1778 tablet_bootstrap.cc:654] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.784019 1848 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.785388 1779 ts_tablet_manager.cc:1397] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.002s
I20260927 23:30:18.787972 1779 raft_consensus.cc:348] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.789233 1779 raft_consensus.cc:374] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.789523 1779 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.790207 1779 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.792749 1779 ts_tablet_manager.cc:1428] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:18.792987 1777 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.793394 1777 ts_tablet_manager.cc:1397] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260927 23:30:18.794381 1779 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.795737 1777 raft_consensus.cc:348] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.796383 1777 raft_consensus.cc:374] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.796639 1777 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.797646 1777 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.799613 1777 ts_tablet_manager.cc:1428] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 23:30:18.800087 1779 tablet_bootstrap.cc:654] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.800428 1777 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.804492 1778 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.805351 1778 ts_tablet_manager.cc:1397] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.007s
I20260927 23:30:18.806948 1777 tablet_bootstrap.cc:654] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.807664 1778 raft_consensus.cc:348] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.808369 1778 raft_consensus.cc:374] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.808645 1778 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.809248 1778 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.812626 1778 ts_tablet_manager.cc:1428] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:30:18.814600 1778 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.817251 1777 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.817886 1777 ts_tablet_manager.cc:1397] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20260927 23:30:18.819706 1778 tablet_bootstrap.cc:654] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.821800 1779 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.822192 1779 ts_tablet_manager.cc:1397] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.000s
I20260927 23:30:18.824939 1779 raft_consensus.cc:348] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.825608 1779 raft_consensus.cc:374] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.825843 1779 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.826452 1779 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.826706 1777 raft_consensus.cc:348] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.827735 1777 raft_consensus.cc:374] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.828037 1777 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.828462 1779 ts_tablet_manager.cc:1428] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 23:30:18.828358 1848 consensus_queue.cc:1035] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.828797 1777 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.829578 1779 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.830911 1777 ts_tablet_manager.cc:1428] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260927 23:30:18.835268 1779 tablet_bootstrap.cc:654] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.844185 1779 tablet_bootstrap.cc:492] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.844590 1779 ts_tablet_manager.cc:1397] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260927 23:30:18.845585 1848 consensus_queue.cc:1035] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.846956 1779 raft_consensus.cc:348] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.847626 1779 raft_consensus.cc:374] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.847886 1779 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.848552 1779 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.850808 1779 ts_tablet_manager.cc:1428] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 23:30:18.851044 1777 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:18.851481 1778 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:18.851606 1779 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.853432 1778 ts_tablet_manager.cc:1397] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.005s
I20260927 23:30:18.855995 1785 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.856374 1778 raft_consensus.cc:348] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.857244 1777 tablet_bootstrap.cc:654] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.857033 1778 raft_consensus.cc:374] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.857510 1779 tablet_bootstrap.cc:654] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.857779 1778 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.858511 1778 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.860642 1778 ts_tablet_manager.cc:1428] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260927 23:30:18.861464 1778 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:18.878237 1779 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:18.878733 1779 ts_tablet_manager.cc:1397] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.011s
I20260927 23:30:18.879320 1778 tablet_bootstrap.cc:654] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.881486 1779 raft_consensus.cc:348] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.882153 1779 raft_consensus.cc:374] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:18.882438 1779 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.883056 1779 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.885036 1779 ts_tablet_manager.cc:1428] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 23:30:18.886407 1779 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:18.887072 1838 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.887490 1838 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.888741 1836 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:18.889784 1838 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.891388 1779 tablet_bootstrap.cc:654] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:18.893023 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.893877 1661 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.894364 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.895387 1736 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.895341 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.897104 1802 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.897567 1802 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.897958 1802 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.903493 1781 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.903956 1781 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.905874 1781 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.907094 1802 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.909564 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:18.910174 1663 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.910835 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.911535 1736 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:18.914175 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:18.912526 1838 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.915611 1838 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.916581 1848 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.917007 1848 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.917335 1848 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.918979 1838 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.919837 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.920455 1660 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.921525 1786 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.921955 1786 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.923354 1663 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.923758 1786 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.925345 1551 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.925513 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.926110 1662 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.926186 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.927476 1906 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.928066 1777 tablet_bootstrap.cc:492] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:18.927821 1906 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.928619 1786 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.928897 1906 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.929036 1786 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:18.929852 1786 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.934297 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:18.934963 1586 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:18.937877 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.938702 1734 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.940384 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.941393 1736 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:18.940765 1802 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.943081 1551 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.943872 1781 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.944726 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.944444 1781 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.945537 1802 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:18.948107 1802 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:18.948141 1781 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.949003 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:18.950433 1628 leader_election.cc:341] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.948967 1848 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.951300 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:18.948534 1802 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:18.953208 1627 leader_election.cc:341] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:18.953764 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:18.955516 1781 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:18.955379 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:18.956588 1586 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.956640 1736 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.959977 1802 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.961860 1802 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.962770 1848 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:18.964138 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:18.964564 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.964725 1661 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.965144 1737 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.966922 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:18.967784 1734 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:18.970938 1661 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.970938 1737 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:18.972118 1586 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:18.973083 1551 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.973412 1628 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:18.974305 1835 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:18.974756 1835 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.976519 1838 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.977237 1777 ts_tablet_manager.cc:1397] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.017s
I20260927 23:30:18.978662 1835 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:18.981082 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:18.983485 1848 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:18.990731 1906 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:18.999989 1838 raft_consensus.cc:686] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:19.001189 1848 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:19.001015 1838 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.001986 1848 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.002861 1777 raft_consensus.cc:348] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.003765 1777 raft_consensus.cc:374] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.004078 1777 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:18.992666 1663 raft_consensus.cc:2379] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:19.025926 1835 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.026769 1734 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:19.026638 1835 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.035988 1736 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.041659 1777 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.048897 1937 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.049743 1937 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.050765 1906 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.057333 1777 ts_tablet_manager.cc:1428] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.065s user 0.008s sys 0.000s
I20260927 23:30:19.058266 1777 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.064605 1835 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:19.064972 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.065923 1663 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:19.067397 1700 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.068853 1822 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.069769 1822 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.070160 1822 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.046440 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:19.082362 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 99831.1us
I20260927 23:30:19.100121 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.101970 1778 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.102656 1778 ts_tablet_manager.cc:1397] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.241s user 0.017s sys 0.014s
I20260927 23:30:19.101289 1585 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:19.110988 1702 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:19.105641 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.112403 1835 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.114591 1734 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.118918 1779 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.119424 1779 ts_tablet_manager.cc:1397] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.233s user 0.071s sys 0.115s
I20260927 23:30:19.120311 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.120910 1661 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.121874 1779 raft_consensus.cc:348] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.121668 1485 catalog_manager.cc:5495] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.122510 1779 raft_consensus.cc:374] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.122778 1779 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.123361 1779 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.124794 1779 ts_tablet_manager.cc:1428] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 23:30:19.125615 1779 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.125952 1734 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.128458 1778 raft_consensus.cc:348] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.131052 1778 raft_consensus.cc:374] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.131294 1778 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.131706 1777 tablet_bootstrap.cc:654] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.132423 1778 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.120911 1937 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.148741 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.149158 1778 ts_tablet_manager.cc:1428] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.046s user 0.004s sys 0.002s
I20260927 23:30:19.149407 1736 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.156235 1778 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:19.157676 1661 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.158303 1482 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.158978 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.160538 1482 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.152784 1822 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:19.164486 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:19.164981 1822 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:19.163622 1663 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.168488 1836 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.168989 1836 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.170197 1552 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.171481 1836 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:19.172976 1935 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.177955 1935 raft_consensus.cc:686] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:19.178772 1935 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.179827 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.177167 1551 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:19.176586 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.183840 1786 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260927 23:30:19.184486 1700 leader_election.cc:341] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.184882 1786 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.185273 1786 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.187017 1835 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:19.183902 1661 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.189400 1779 tablet_bootstrap.cc:654] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.193142 1786 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.194971 1777 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.195379 1777 ts_tablet_manager.cc:1397] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.137s user 0.014s sys 0.005s
I20260927 23:30:19.191039 1935 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.201581 1938 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:19.203094 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.200754 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:19.205246 1586 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.206247 1661 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:19.207657 1700 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.208389 1835 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.209918 1835 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:30:19.211187 1702 leader_election.cc:341] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.211624 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:19.213364 1835 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.219277 1893 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:19.225155 1786 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.225865 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.229671 1482 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.226475 1662 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.231662 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.232460 1737 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.238462 1662 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.239336 1779 tablet_bootstrap.cc:492] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.239806 1779 ts_tablet_manager.cc:1397] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.004s
I20260927 23:30:19.239590 1551 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:19.240629 1778 tablet_bootstrap.cc:654] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.241541 1935 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.242038 1935 raft_consensus.cc:686] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:19.242758 1779 raft_consensus.cc:348] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.241727 1777 raft_consensus.cc:348] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.243808 1893 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.243479 1935 consensus_queue.cc:237] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.243973 1777 raft_consensus.cc:374] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.244587 1777 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.243511 1779 raft_consensus.cc:374] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.245040 1779 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.245613 1779 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.244311 1893 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.248243 1779 ts_tablet_manager.cc:1428] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 23:30:19.249419 1779 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.250397 1893 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:19.246428 1777 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.251266 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.252837 1663 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:19.254590 1700 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.258661 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.260600 1906 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:19.260759 1702 leader_election.cc:341] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.261196 1906 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.262130 1586 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:19.263214 1906 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.263705 1483 catalog_manager.cc:5495] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:19.267084 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.267689 1663 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.270020 1835 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.270521 1835 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.270919 1835 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.271009 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.265237 1737 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.271680 1734 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.273470 1777 ts_tablet_manager.cc:1428] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.078s user 0.008s sys 0.000s
I20260927 23:30:19.274302 1777 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.280300 1777 tablet_bootstrap.cc:654] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.281018 1482 catalog_manager.cc:5495] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.285951 1552 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.287509 1906 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.287889 1906 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.288209 1906 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.290575 1835 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.297108 1778 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.297303 1779 tablet_bootstrap.cc:654] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.297537 1778 ts_tablet_manager.cc:1397] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.008s
I20260927 23:30:19.299124 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:19.300379 1702 leader_election.cc:341] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.301174 1835 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:19.302433 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.307204 1906 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.304728 1662 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.309130 1906 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.311059 1777 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.311257 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.311493 1777 ts_tablet_manager.cc:1397] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.001s
I20260927 23:30:19.311774 1663 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.313858 1777 raft_consensus.cc:348] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.314529 1777 raft_consensus.cc:374] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.314785 1777 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.315626 1662 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:19.315388 1777 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.316570 1700 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:19.317265 1778 raft_consensus.cc:348] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.317785 1835 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.318347 1835 raft_consensus.cc:686] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:19.319018 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.319150 1835 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.319597 1736 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.320050 1777 ts_tablet_manager.cc:1428] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:30:19.321004 1777 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.327389 1777 tablet_bootstrap.cc:654] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.330843 1736 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.331707 1779 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.317973 1778 raft_consensus.cc:374] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.332113 1779 ts_tablet_manager.cc:1397] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.000s
I20260927 23:30:19.332082 1778 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.331988 1552 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.332850 1778 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.334415 1779 raft_consensus.cc:348] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.334903 1778 ts_tablet_manager.cc:1428] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.037s user 0.005s sys 0.003s
I20260927 23:30:19.335053 1779 raft_consensus.cc:374] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.335309 1779 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.332901 1906 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.335877 1779 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.336119 1935 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.337345 1778 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:19.318003 1663 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.338693 1906 raft_consensus.cc:686] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:19.339854 1906 consensus_queue.cc:237] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.359269 1779 ts_tablet_manager.cc:1428] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.027s user 0.007s sys 0.001s
I20260927 23:30:19.360369 1779 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.365289 1777 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.365722 1779 tablet_bootstrap.cc:654] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.365782 1777 ts_tablet_manager.cc:1397] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.015s
I20260927 23:30:19.370222 1778 tablet_bootstrap.cc:654] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.370043 1483 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:19.374212 1777 raft_consensus.cc:348] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.380995 1777 raft_consensus.cc:374] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.381273 1777 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.381870 1777 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.388792 1777 ts_tablet_manager.cc:1428] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260927 23:30:19.389810 1777 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.392057 1848 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.392999 1485 catalog_manager.cc:5495] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.402273 1777 tablet_bootstrap.cc:654] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.402698 1935 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.412173 1935 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.430078 1779 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.430667 1779 ts_tablet_manager.cc:1397] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.071s user 0.020s sys 0.000s
I20260927 23:30:19.438225 1779 raft_consensus.cc:348] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.439263 1779 raft_consensus.cc:374] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.439653 1779 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.440526 1779 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:19.448768 1954 log.cc:924] Time spent T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260927 23:30:19.456197 1935 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.456701 1935 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.458752 1779 ts_tablet_manager.cc:1428] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260927 23:30:19.460028 1779 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.461809 1935 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.465404 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.466462 1778 tablet_bootstrap.cc:492] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.466866 1778 ts_tablet_manager.cc:1397] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.130s user 0.013s sys 0.006s
I20260927 23:30:19.468604 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:19.474049 1552 leader_election.cc:341] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.475638 1662 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.476742 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:19.480208 1786 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.482232 1786 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.482577 1786 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.481674 1778 raft_consensus.cc:348] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.486177 1778 raft_consensus.cc:374] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.486627 1778 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.487579 1778 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.495059 1778 ts_tablet_manager.cc:1428] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260927 23:30:19.495970 1778 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:19.502796 1786 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.505625 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.506191 1663 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.509032 1779 tablet_bootstrap.cc:654] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.509409 1777 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.509836 1777 ts_tablet_manager.cc:1397] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.120s user 0.010s sys 0.018s
I20260927 23:30:19.511515 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.513315 1778 tablet_bootstrap.cc:654] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:19.513614 1552 leader_election.cc:341] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.513830 1854 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.514178 1786 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.517400 1777 raft_consensus.cc:348] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.518805 1777 raft_consensus.cc:374] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.519071 1777 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.519531 1777 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.523762 1663 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:19.527912 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:19.529763 1937 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.530191 1937 raft_consensus.cc:686] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:19.530980 1937 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.533357 1854 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.533847 1854 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.534374 1935 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.536527 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.537637 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.537361 1854 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.538204 1736 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:19.539686 1777 ts_tablet_manager.cc:1428] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.030s user 0.005s sys 0.003s
I20260927 23:30:19.540529 1777 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.542760 1627 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.543602 1854 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.544067 1854 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:30:19.546260 1628 leader_election.cc:341] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.546861 1854 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.561264 1779 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.561709 1779 ts_tablet_manager.cc:1397] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.005s
I20260927 23:30:19.564755 1854 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.566955 1483 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.567632 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.568190 1736 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.569458 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:19.573382 1781 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:19.573935 1628 leader_election.cc:341] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.573829 1781 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.575762 1781 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.577634 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.578244 1586 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:19.578717 1778 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.579183 1778 ts_tablet_manager.cc:1397] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.083s user 0.017s sys 0.004s
I20260927 23:30:19.579948 1628 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:19.580760 1853 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.581084 1853 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.581380 1853 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.581753 1736 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:19.581907 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.582912 1627 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:19.583684 1781 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.584178 1781 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:19.584770 1779 raft_consensus.cc:348] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.585496 1779 raft_consensus.cc:374] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.584905 1781 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.585765 1779 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.586422 1779 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.588327 1777 tablet_bootstrap.cc:654] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.588485 1779 ts_tablet_manager.cc:1428] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.027s user 0.004s sys 0.002s
I20260927 23:30:19.590405 1779 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.590767 1854 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.595556 1779 tablet_bootstrap.cc:654] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.588469 1853 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.598047 1778 raft_consensus.cc:348] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.598745 1778 raft_consensus.cc:374] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.598987 1778 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.599368 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:19.600358 1586 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.600059 1778 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:19.601600 1627 leader_election.cc:341] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.606057 1778 ts_tablet_manager.cc:1428] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260927 23:30:19.606949 1778 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:19.612221 1854 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.612838 1854 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.613842 1779 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.614257 1779 ts_tablet_manager.cc:1397] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260927 23:30:19.615973 1854 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.618619 1777 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.618870 1779 raft_consensus.cc:348] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.619252 1777 ts_tablet_manager.cc:1397] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.079s user 0.024s sys 0.006s
I20260927 23:30:19.619421 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:19.619655 1779 raft_consensus.cc:374] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.620108 1779 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.620745 1779 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.618649 1778 tablet_bootstrap.cc:654] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.622367 1586 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:19.622128 1777 raft_consensus.cc:348] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:19.623183 1628 leader_election.cc:341] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.621676 1853 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.624015 1777 raft_consensus.cc:374] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.624404 1777 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.625085 1628 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:19.625136 1777 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.628986 1937 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:19.631325 1853 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.632396 1853 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:19.634373 1777 ts_tablet_manager.cc:1428] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20260927 23:30:19.635466 1777 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.633548 1853 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.630587 1779 ts_tablet_manager.cc:1428] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260927 23:30:19.641327 1779 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.648144 1778 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.648272 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.649039 1736 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:19.649575 1778 ts_tablet_manager.cc:1397] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.010s
I20260927 23:30:19.649240 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.653322 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:30:19.654822 1627 leader_election.cc:341] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.655002 1778 raft_consensus.cc:348] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.655704 1778 raft_consensus.cc:374] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.655983 1778 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.656728 1937 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.657699 1937 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.659368 1848 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.661652 1848 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.662068 1848 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.662706 1937 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.663357 1937 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.670861 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.694407 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:19.699407 1551 leader_election.cc:341] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.724447 1969 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.725162 1969 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.727869 1969 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:19.700294 1552 leader_election.cc:341] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.761627 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:19.765606 1627 leader_election.cc:341] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.758826 1552 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:19.791393 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 91150us
I20260927 23:30:19.793252 1786 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:19.828806 1779 tablet_bootstrap.cc:700] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Time spent opening tablet: real 0.136s user 0.004s sys 0.012s
I20260927 23:30:19.831729 1779 tablet_bootstrap.cc:654] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.826287 1778 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.832980 1777 tablet_bootstrap.cc:654] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.841553 1778 ts_tablet_manager.cc:1428] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.191s user 0.044s sys 0.095s
I20260927 23:30:19.842494 1778 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:19.890638 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:19.892102 1628 leader_election.cc:341] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:19.891933 1836 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.892371 1786 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.892568 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:19.893630 1853 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:19.898242 1937 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:19.884944 1848 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.905418 1482 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.911566 1906 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.912112 1906 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.913996 1906 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.914404 1848 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.915298 1779 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:19.915771 1779 ts_tablet_manager.cc:1397] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.275s user 0.031s sys 0.041s
I20260927 23:30:19.924310 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:19.925025 1734 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.930783 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:19.931397 1661 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.931834 1777 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:19.932536 1777 ts_tablet_manager.cc:1397] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.297s user 0.060s sys 0.108s
I20260927 23:30:19.932514 1551 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:19.911778 1778 tablet_bootstrap.cc:654] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.921602 1779 raft_consensus.cc:348] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.934293 1906 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:19.934659 1906 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:19.934978 1906 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.934095 1779 raft_consensus.cc:374] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.935552 1779 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.942303 1777 raft_consensus.cc:348] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.943017 1777 raft_consensus.cc:374] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:19.943260 1777 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:19.945151 1483 catalog_manager.cc:5495] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:19.946939 1734 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:19.945739 1906 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.949295 1906 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:19.936218 1779 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.949584 1937 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.951891 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:19.952430 1661 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:19.943811 1777 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.955317 1835 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:19.955732 1893 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:19.958751 1777 ts_tablet_manager.cc:1428] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260927 23:30:19.959772 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:19.960602 1848 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:19.961081 1848 raft_consensus.cc:686] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:19.961923 1848 consensus_queue.cc:237] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.959968 1777 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:19.968327 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:19.969213 1736 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:19.971920 1779 ts_tablet_manager.cc:1428] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.054s user 0.003s sys 0.002s
I20260927 23:30:19.974458 1779 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:19.981921 1853 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:19.983389 1853 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:19.985550 1777 tablet_bootstrap.cc:654] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:19.992193 1778 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:19.992620 1778 ts_tablet_manager.cc:1397] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.150s user 0.021s sys 0.008s
I20260927 23:30:19.988395 1853 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:20.000247 1702 consensus_peers.cc:487] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Couldn't send request to peer 0fe98f486bab4d0b9c421c145d83f1ce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:20.007238 1976 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.006071 1661 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:20.007745 1976 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.008513 1551 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:20.010046 1937 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.011150 1835 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.013794 1778 raft_consensus.cc:348] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.015069 1778 raft_consensus.cc:374] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.015419 1778 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.015966 1778 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.010243 1976 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.011754 1835 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.019681 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.022872 1835 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.023458 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:20.024076 1733 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.024801 1778 ts_tablet_manager.cc:1428] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.032s user 0.006s sys 0.002s
I20260927 23:30:20.025440 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.025781 1778 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.012346 1937 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:20.027060 1937 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.021006 1483 catalog_manager.cc:5495] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.022889 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:20.035789 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.037792 1906 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:20.045680 1733 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:20.046999 1779 tablet_bootstrap.cc:654] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:20.048657 1627 leader_election.cc:341] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.050156 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.052750 1835 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.053261 1835 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.055219 1835 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:20.055482 1700 leader_election.cc:341] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.064553 1482 catalog_manager.cc:5495] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.070536 1778 tablet_bootstrap.cc:654] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.074494 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.078615 1937 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.079092 1937 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.080973 1937 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.082268 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.084553 1777 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.085170 1777 ts_tablet_manager.cc:1397] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.125s user 0.014s sys 0.002s
W20260927 23:30:20.086381 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 48630.9us
I20260927 23:30:20.090791 1777 raft_consensus.cc:348] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.091485 1777 raft_consensus.cc:374] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.091742 1777 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
W20260927 23:30:20.093029 1628 leader_election.cc:341] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.095057 1733 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.093576 1777 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.108825 1777 ts_tablet_manager.cc:1428] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260927 23:30:20.112104 1779 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.112629 1779 ts_tablet_manager.cc:1397] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.138s user 0.015s sys 0.000s
I20260927 23:30:20.113092 1848 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.115192 1779 raft_consensus.cc:348] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:20.117666 1628 leader_election.cc:341] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.118443 1628 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:20.120326 1781 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.121368 1779 raft_consensus.cc:374] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.121742 1779 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.126691 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.127517 1779 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.131129 1779 ts_tablet_manager.cc:1428] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260927 23:30:20.131003 1786 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:20.132089 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.133370 1587 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:20.133836 1779 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.134526 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.135643 1778 tablet_bootstrap.cc:492] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.136103 1778 ts_tablet_manager.cc:1397] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.005s
I20260927 23:30:20.137313 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.141070 1777 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.145244 1779 tablet_bootstrap.cc:654] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.148869 1778 raft_consensus.cc:348] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.149576 1778 raft_consensus.cc:374] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.151049 1778 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.152479 1786 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.153038 1702 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:20.153345 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:20.154034 1786 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.154173 1587 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:20.154712 1702 leader_election.cc:341] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.155144 1702 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:20.156248 1702 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:20.158084 1778 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.156289 1733 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.160072 1835 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.158905 1893 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.164937 1778 ts_tablet_manager.cc:1428] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260927 23:30:20.165789 1778 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.168655 1937 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.170810 1937 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.178507 1937 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.182595 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.183434 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.185175 1552 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:20.187084 1853 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:20.187111 1786 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:20.189850 1779 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.190332 1779 ts_tablet_manager.cc:1397] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.010s
I20260927 23:30:20.194988 1779 raft_consensus.cc:348] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.195667 1779 raft_consensus.cc:374] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.195917 1779 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.196485 1779 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.202237 1778 tablet_bootstrap.cc:654] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.206682 1779 ts_tablet_manager.cc:1428] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260927 23:30:20.223570 1779 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.227784 1853 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:20.243046 1777 tablet_bootstrap.cc:654] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.252770 1779 tablet_bootstrap.cc:654] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.257604 1976 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.264618 1853 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 23:30:20.270157 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:20.287580 1853 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.288233 1853 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.290787 1853 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.291230 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.291887 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.293387 1737 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:20.294502 1778 tablet_bootstrap.cc:492] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.294974 1778 ts_tablet_manager.cc:1397] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.129s user 0.020s sys 0.014s
I20260927 23:30:20.303155 1777 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.303637 1777 ts_tablet_manager.cc:1397] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.163s user 0.011s sys 0.012s
I20260927 23:30:20.294629 1627 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:30:20.306434 1628 leader_election.cc:341] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.307677 1778 raft_consensus.cc:348] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.308408 1778 raft_consensus.cc:374] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.308648 1778 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.309329 1778 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.310509 1848 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.310952 1848 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.311292 1848 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.311771 1778 ts_tablet_manager.cc:1428] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260927 23:30:20.312985 1778 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.322026 1777 raft_consensus.cc:348] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.328346 1777 raft_consensus.cc:374] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.328685 1777 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.323125 1848 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.329331 1777 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.331154 1777 ts_tablet_manager.cc:1428] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260927 23:30:20.331459 1848 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.331987 1777 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.332691 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:20.333076 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:20.328058 1779 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.325973 1778 tablet_bootstrap.cc:654] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.333238 1737 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.334177 1779 ts_tablet_manager.cc:1397] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.111s user 0.020s sys 0.020s
I20260927 23:30:20.336838 1779 raft_consensus.cc:348] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.337524 1779 raft_consensus.cc:374] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.337801 1779 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.337973 1777 tablet_bootstrap.cc:654] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.339640 1779 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:20.341220 1628 leader_election.cc:341] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.343644 1779 ts_tablet_manager.cc:1428] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 23:30:20.344512 1779 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.352406 1779 tablet_bootstrap.cc:654] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.354025 1737 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:20.361464 1778 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.362502 1778 ts_tablet_manager.cc:1397] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.003s
I20260927 23:30:20.362973 1627 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.364187 1848 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.364677 1848 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:20.365442 1848 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.370824 1779 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.371272 1779 ts_tablet_manager.cc:1397] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20260927 23:30:20.372818 1778 raft_consensus.cc:348] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.373308 1778 raft_consensus.cc:374] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.373579 1778 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.373617 1779 raft_consensus.cc:348] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.374221 1779 raft_consensus.cc:374] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.374655 1779 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.374197 1778 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.374917 1585 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.375239 1779 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.377717 1779 ts_tablet_manager.cc:1428] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 23:30:20.378396 1778 ts_tablet_manager.cc:1428] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260927 23:30:20.379096 1778 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.382468 1777 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.382889 1777 ts_tablet_manager.cc:1397] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.051s user 0.025s sys 0.007s
I20260927 23:30:20.385042 1777 raft_consensus.cc:348] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.385696 1777 raft_consensus.cc:374] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.385962 1777 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.386621 1777 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.388473 1777 ts_tablet_manager.cc:1428] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 23:30:20.389227 1777 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.390369 1483 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.393005 1779 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.394115 1777 tablet_bootstrap.cc:654] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.398674 1779 tablet_bootstrap.cc:654] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.407159 1778 tablet_bootstrap.cc:654] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.407981 1786 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.408608 1786 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.413571 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.414542 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:20.415197 1551 leader_election.cc:341] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.415382 1737 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:20.413573 1786 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.416894 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:20.418263 1786 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.418735 1786 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.419134 1786 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.421986 1779 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.422487 1779 ts_tablet_manager.cc:1397] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.003s
I20260927 23:30:20.422863 1778 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.423308 1778 ts_tablet_manager.cc:1397] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.005s
I20260927 23:30:20.426180 1587 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.429008 1935 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.435525 1848 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:20.437706 1778 raft_consensus.cc:348] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.439492 1778 raft_consensus.cc:374] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.439782 1778 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.440382 1778 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.426040 1786 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.444510 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.444510 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:20.446440 1737 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.448485 1786 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:20.449656 1551 leader_election.cc:341] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.454309 1737 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:20.455377 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:20.457175 1786 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.457584 1786 raft_consensus.cc:686] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:20.458249 1786 consensus_queue.cc:237] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.460000 1777 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.460439 1777 ts_tablet_manager.cc:1397] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.028s
I20260927 23:30:20.462723 1777 raft_consensus.cc:348] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.463371 1777 raft_consensus.cc:374] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.463622 1777 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.464188 1777 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.464829 1778 ts_tablet_manager.cc:1428] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.041s user 0.005s sys 0.002s
I20260927 23:30:20.466091 1777 ts_tablet_manager.cc:1428] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 23:30:20.471372 1893 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.471974 1893 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.475255 1893 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.476282 1777 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.475587 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.476877 1588 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:20.478631 1702 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:20.480088 1893 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.480475 1893 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.480860 1893 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.481809 1777 tablet_bootstrap.cc:654] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.487581 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.488319 1662 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:20.456485 1779 raft_consensus.cc:348] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.496266 1779 raft_consensus.cc:374] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.497128 1779 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.499289 1777 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.499833 1976 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.500396 1777 ts_tablet_manager.cc:1397] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.003s
I20260927 23:30:20.500422 1976 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.502535 1976 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.502875 1777 raft_consensus.cc:348] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.503537 1777 raft_consensus.cc:374] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.503798 1777 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.503952 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.504391 1777 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:20.508093 1702 leader_election.cc:341] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.510309 1854 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.511312 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.512243 1777 ts_tablet_manager.cc:1428] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260927 23:30:20.513036 1777 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.515702 1935 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.516363 1935 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.519296 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.519861 1661 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:20.512008 1662 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:20.520895 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:20.518877 1935 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.523236 1935 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.523592 1935 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.523949 1935 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.524588 1777 tablet_bootstrap.cc:654] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.525215 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.528054 1835 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.528429 1835 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.528775 1835 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.498710 1779 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.532333 1779 ts_tablet_manager.cc:1428] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.110s user 0.004s sys 0.002s
I20260927 23:30:20.533381 1779 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.534260 1935 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.535832 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.537125 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.537703 1777 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.537714 1662 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.539722 1777 ts_tablet_manager.cc:1397] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.006s
I20260927 23:30:20.538602 1778 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
W20260927 23:30:20.538465 1552 leader_election.cc:341] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.539031 1935 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.543577 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:20.545205 1552 leader_election.cc:341] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.545224 1777 raft_consensus.cc:348] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.546029 1777 raft_consensus.cc:374] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.546284 1777 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.546888 1777 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.548812 1777 ts_tablet_manager.cc:1428] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260927 23:30:20.549655 1777 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.558364 1662 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:20.556002 1976 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.561056 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:20.560868 1976 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.562822 1987 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.563789 1777 tablet_bootstrap.cc:654] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.566092 1976 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.567448 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.556273 1893 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.568661 1987 raft_consensus.cc:686] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:20.568572 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:20.569512 1702 leader_election.cc:341] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.569799 1661 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:20.570078 1893 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.567865 1835 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.569540 1987 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.571159 1935 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 23:30:20.571681 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.575451 1835 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.575690 1661 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.576112 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:20.579039 1702 leader_election.cc:341] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.571931 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:20.579829 1585 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.572319 1700 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.582908 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.583447 1662 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.586972 1585 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:20.588055 1661 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:20.588022 1702 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:20.588917 1893 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.589267 1893 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.589574 1893 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.591041 1976 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.593309 1779 tablet_bootstrap.cc:654] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.590692 1777 tablet_bootstrap.cc:492] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.596859 1777 ts_tablet_manager.cc:1397] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.000s
I20260927 23:30:20.597622 1976 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:20.600214 1777 raft_consensus.cc:348] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.600899 1777 raft_consensus.cc:374] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.601150 1777 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.598515 1976 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.603751 1662 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:20.604898 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.604626 1777 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.605820 1994 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.606247 1994 raft_consensus.cc:686] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:20.607002 1777 ts_tablet_manager.cc:1428] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 23:30:20.607079 1994 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.607955 1777 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.608614 1778 tablet_bootstrap.cc:654] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.611359 1482 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.613001 1893 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.613042 1482 catalog_manager.cc:5495] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.618788 1779 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.619258 1779 ts_tablet_manager.cc:1397] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.011s
I20260927 23:30:20.621496 1777 tablet_bootstrap.cc:654] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.629649 1893 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.631901 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:20.633283 1702 leader_election.cc:341] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.635959 1779 raft_consensus.cc:348] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.638635 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.641788 1662 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.641013 1483 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:20.646118 1483 catalog_manager.cc:5495] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:20.648983 1779 raft_consensus.cc:374] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.649336 1779 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.650060 1779 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.652585 1779 ts_tablet_manager.cc:1428] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260927 23:30:20.652669 1777 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.651326 1662 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:20.653414 1777 ts_tablet_manager.cc:1397] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.004s
I20260927 23:30:20.654074 1700 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.655232 1994 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.655709 1994 raft_consensus.cc:686] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:20.655767 1777 raft_consensus.cc:348] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.656606 1777 raft_consensus.cc:374] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.656883 1777 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.657202 1994 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.663769 1778 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.663399 1779 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.663241 1777 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.664229 1778 ts_tablet_manager.cc:1397] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.018s
I20260927 23:30:20.671942 1777 ts_tablet_manager.cc:1428] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.018s user 0.008s sys 0.004s
I20260927 23:30:20.675315 1777 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.677258 1778 raft_consensus.cc:348] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.677983 1778 raft_consensus.cc:374] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.678251 1778 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.678978 1778 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.681066 1778 ts_tablet_manager.cc:1428] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20260927 23:30:20.682111 1778 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.682154 1854 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.682646 1854 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.683513 1779 tablet_bootstrap.cc:654] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.685177 1854 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.685475 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.686076 1585 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:20.686597 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.686944 1777 tablet_bootstrap.cc:654] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.687140 1628 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:30:20.687945 1627 leader_election.cc:341] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.687945 1854 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.689040 1854 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.689355 1854 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.693301 1778 tablet_bootstrap.cc:654] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.703596 1779 tablet_bootstrap.cc:492] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.703074 1854 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.704169 1779 ts_tablet_manager.cc:1397] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.005s
I20260927 23:30:20.706544 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:20.707333 1585 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.709131 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:20.709870 1854 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.715004 1585 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:20.716099 1628 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:20.717037 1854 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.717531 1854 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:20.717799 1482 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:20.718497 1854 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.724714 1777 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.725776 1777 ts_tablet_manager.cc:1397] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.005s
I20260927 23:30:20.728524 1777 raft_consensus.cc:348] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.707079 1779 raft_consensus.cc:348] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.729840 1779 raft_consensus.cc:374] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.730115 1779 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.733301 1778 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.731000 1779 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.733829 1778 ts_tablet_manager.cc:1397] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.020s
I20260927 23:30:20.734058 1733 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.736591 1778 raft_consensus.cc:348] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.737327 1778 raft_consensus.cc:374] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.737632 1778 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.738564 1778 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.737607 1482 catalog_manager.cc:5495] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:20.741298 1778 ts_tablet_manager.cc:1428] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:20.741233 1733 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:20.742517 1778 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.730278 1777 raft_consensus.cc:374] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.745816 1777 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.746510 1777 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.748066 1777 ts_tablet_manager.cc:1428] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260927 23:30:20.748901 1777 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.749233 1778 tablet_bootstrap.cc:654] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.745472 1779 ts_tablet_manager.cc:1428] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260927 23:30:20.752961 1779 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.755944 1777 tablet_bootstrap.cc:654] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.760249 1779 tablet_bootstrap.cc:654] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.760727 1778 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.761293 1778 ts_tablet_manager.cc:1397] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20260927 23:30:20.764326 1778 raft_consensus.cc:348] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.765002 1778 raft_consensus.cc:374] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.765304 1778 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.765859 1778 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.768033 1778 ts_tablet_manager.cc:1428] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 23:30:20.768810 1778 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.774446 1778 tablet_bootstrap.cc:654] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.778682 1777 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.779381 1777 ts_tablet_manager.cc:1397] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.000s
I20260927 23:30:20.782198 1781 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.787941 1777 raft_consensus.cc:348] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.789126 1777 raft_consensus.cc:374] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.789547 1777 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.794167 1779 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.796010 1779 ts_tablet_manager.cc:1397] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.008s
I20260927 23:30:20.794703 1777 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.800604 1777 ts_tablet_manager.cc:1428] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260927 23:30:20.802261 1779 raft_consensus.cc:348] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.803885 1779 raft_consensus.cc:374] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.804175 1779 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.804821 1779 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.805943 1778 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.806439 1778 ts_tablet_manager.cc:1397] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.024s
I20260927 23:30:20.806939 1779 ts_tablet_manager.cc:1428] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260927 23:30:20.807767 1779 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.808861 1778 raft_consensus.cc:348] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.809592 1778 raft_consensus.cc:374] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.809890 1778 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.810616 1778 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.813978 1778 ts_tablet_manager.cc:1428] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:20.814424 1779 tablet_bootstrap.cc:654] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.814879 1778 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:20.820482 1976 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.820952 1976 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.823246 1777 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.824100 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:20.824736 1662 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:20.825986 1700 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:20.826829 1976 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.827491 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.827728 1976 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.828202 1976 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.828590 1976 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:20.828830 1702 leader_election.cc:341] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.829617 1777 tablet_bootstrap.cc:654] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.830714 1778 tablet_bootstrap.cc:654] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.841619 1976 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.845423 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:20.847601 1702 leader_election.cc:341] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.848886 1585 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.859058 1853 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.860628 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:20.861490 1662 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.861601 1779 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.862407 1779 ts_tablet_manager.cc:1397] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.055s user 0.020s sys 0.020s
I20260927 23:30:20.864619 1779 raft_consensus.cc:348] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.865272 1779 raft_consensus.cc:374] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.865506 1779 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.866663 1779 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.868893 1779 ts_tablet_manager.cc:1428] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 23:30:20.869626 1779 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:20.875385 1779 tablet_bootstrap.cc:654] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:20.885736 1661 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.892184 1848 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.892820 1848 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.892333 1781 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.894692 1778 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:20.895865 1778 ts_tablet_manager.cc:1397] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.081s user 0.023s sys 0.022s
I20260927 23:30:20.899701 1781 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.903082 1781 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.901539 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.904919 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.905761 1733 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:20.898504 1976 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:20.892227 1935 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260927 23:30:20.907949 1937 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:20.894735 1848 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.911129 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.903883 1585 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:20.920717 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.915258 1627 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:20.932085 1848 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:20.934144 1848 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.935964 1848 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:20.936079 1777 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:20.936265 1662 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:20.936528 1777 ts_tablet_manager.cc:1397] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.113s user 0.009s sys 0.001s
W20260927 23:30:20.938514 1627 leader_election.cc:341] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.938643 1848 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.939080 1848 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.938901 1777 raft_consensus.cc:348] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.939596 1777 raft_consensus.cc:374] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.939894 1777 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.939808 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:20.940539 1735 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
W20260927 23:30:20.941143 1628 leader_election.cc:341] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.941560 1628 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:20.942272 1700 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:20.942716 1627 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:20.940801 1777 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.943169 1976 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:20.943416 1853 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:20.943697 1854 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:20.944052 1854 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:20.944371 1854 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.945070 1777 ts_tablet_manager.cc:1428] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260927 23:30:20.943706 1976 raft_consensus.cc:686] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:20.949584 1779 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:20.950162 1777 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:20.949750 1976 consensus_queue.cc:237] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.942148 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:20.939442 1848 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:20.952750 1779 ts_tablet_manager.cc:1397] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.083s user 0.028s sys 0.027s
W20260927 23:30:20.966308 1628 leader_election.cc:341] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:20.994110 1778 raft_consensus.cc:348] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.994865 1778 raft_consensus.cc:374] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.995105 1779 raft_consensus.cc:348] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.995750 1779 raft_consensus.cc:374] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:20.996001 1779 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.996539 1779 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:20.995121 1778 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:20.999154 1778 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.004947 1779 ts_tablet_manager.cc:1428] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20260927 23:30:21.006206 1777 tablet_bootstrap.cc:654] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.009874 1778 ts_tablet_manager.cc:1428] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.114s user 0.006s sys 0.000s
I20260927 23:30:21.014530 1778 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.016050 1779 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.047246 1588 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.043102 1854 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.050509 1854 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.062188 1483 catalog_manager.cc:5495] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:21.064945 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.065549 1735 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.075981 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.077311 1778 tablet_bootstrap.cc:654] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.099325 1779 tablet_bootstrap.cc:654] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.101630 1735 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.087718 1976 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.102849 1976 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.107340 1627 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.110692 1893 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.112048 1976 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.114841 1854 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.114439 1848 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.115346 1854 raft_consensus.cc:686] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:21.116585 1854 consensus_queue.cc:237] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.117187 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.119807 1987 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.122973 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.123555 1737 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.123291 1987 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.125193 1987 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:21.122306 1702 leader_election.cc:341] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.127256 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:21.128480 1552 leader_election.cc:341] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.117262 1848 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:21.132611 1552 consensus_peers.cc:487] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Couldn't send request to peer a2a3317e8da742d9aefe4c251abd4566. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:21.133081 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.139029 1777 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
W20260927 23:30:21.139228 1551 leader_election.cc:341] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.139492 1777 ts_tablet_manager.cc:1397] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.015s
I20260927 23:30:21.139638 1551 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.141093 1935 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 23:30:21.142752 1628 leader_election.cc:341] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.147696 1778 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.148136 1778 ts_tablet_manager.cc:1397] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.134s user 0.013s sys 0.017s
I20260927 23:30:21.147794 1777 raft_consensus.cc:348] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.148748 1777 raft_consensus.cc:374] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.149107 1777 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.149427 1737 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.150552 1854 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.150985 1627 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.151130 1854 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.149857 1777 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.153004 2043 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.154032 1777 ts_tablet_manager.cc:1428] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20260927 23:30:21.155193 1777 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.157066 2043 raft_consensus.cc:686] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:21.160043 1778 raft_consensus.cc:348] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.160585 1778 raft_consensus.cc:374] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.160816 1778 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.162310 1778 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.161533 2043 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.163792 1482 catalog_manager.cc:5495] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:21.164785 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.165421 1587 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.164683 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.158141 1854 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.171921 1777 tablet_bootstrap.cc:654] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.175174 1737 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:21.176607 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.177474 1854 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.177743 1587 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.177881 1854 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.178375 1854 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.178503 1779 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:21.179270 1779 ts_tablet_manager.cc:1397] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.164s user 0.012s sys 0.007s
I20260927 23:30:21.180485 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.182466 1779 raft_consensus.cc:348] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.184455 1779 raft_consensus.cc:374] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.184808 1779 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.185591 1779 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.191069 1786 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.192255 1854 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.193521 1779 ts_tablet_manager.cc:1428] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260927 23:30:21.194587 1779 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.197548 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.198441 1662 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:21.201632 1778 ts_tablet_manager.cc:1428] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.053s user 0.003s sys 0.003s
I20260927 23:30:21.201721 1976 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.203413 1893 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.206215 1893 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:21.207943 1778 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.217342 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.217972 1737 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.228421 1737 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.229254 1587 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.230257 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.231743 2044 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.234881 1700 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:21.235255 1893 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.235683 1893 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.210844 1854 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.236902 1994 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.238083 1835 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:21.232182 2044 raft_consensus.cc:686] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:21.232137 1483 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:21.239315 1893 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.246618 2044 consensus_queue.cc:237] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.247864 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:21.253023 1702 leader_election.cc:341] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:21.252758 1628 leader_election.cc:341] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.260931 2043 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.266644 1778 tablet_bootstrap.cc:654] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.263119 2043 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.269986 1994 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.270457 1994 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.271546 2043 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.272274 1994 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.282888 1779 tablet_bootstrap.cc:654] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.279853 1854 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260927 23:30:21.287865 1482 catalog_manager.cc:5495] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:21.290953 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.295403 1777 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.295791 1777 ts_tablet_manager.cc:1397] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.008s
W20260927 23:30:21.296780 1627 leader_election.cc:341] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.298725 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.301787 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.303243 1777 raft_consensus.cc:348] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.304970 1777 raft_consensus.cc:374] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.306553 1777 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.310765 1777 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.318823 1777 ts_tablet_manager.cc:1428] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20260927 23:30:21.319653 1777 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.321460 1976 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.323925 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.330251 1777 tablet_bootstrap.cc:654] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.332146 2041 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:21.343139 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.350857 1660 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:21.351776 2041 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.352272 2041 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.354450 2041 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.363619 1778 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.364667 1778 ts_tablet_manager.cc:1397] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.162s user 0.013s sys 0.005s
I20260927 23:30:21.367233 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.363991 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.374850 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.374941 1700 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:21.377549 1893 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.377915 1893 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.378253 1893 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.380180 1777 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.380646 1777 ts_tablet_manager.cc:1397] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.005s
I20260927 23:30:21.377942 1700 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:21.391112 1835 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 23:30:21.384244 1627 leader_election.cc:341] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:21.391780 1628 leader_election.cc:341] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.393239 1777 raft_consensus.cc:348] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.394860 1777 raft_consensus.cc:374] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.395265 1777 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.398465 1777 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.414997 1779 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:21.415482 1779 ts_tablet_manager.cc:1397] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.221s user 0.015s sys 0.010s
I20260927 23:30:21.415606 1935 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.416100 1935 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.418672 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.417780 1779 raft_consensus.cc:348] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:21.407414 1628 leader_election.cc:341] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.419297 1662 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:21.420261 1628 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.420594 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.419965 1779 raft_consensus.cc:374] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.421487 1779 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.421715 1551 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:21.419281 1935 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.422125 1779 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.422663 1937 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.423399 1937 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.423521 1735 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:21.423739 1937 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.426160 1779 ts_tablet_manager.cc:1428] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260927 23:30:21.422662 1778 raft_consensus.cc:348] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.426646 1778 raft_consensus.cc:374] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.426906 1778 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.427511 2043 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.428025 1778 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.431375 1778 ts_tablet_manager.cc:1428] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.066s user 0.001s sys 0.004s
I20260927 23:30:21.432192 1778 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.433267 1777 ts_tablet_manager.cc:1428] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.052s user 0.009s sys 0.000s
I20260927 23:30:21.433984 1777 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.436054 1779 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
W20260927 23:30:21.430284 1628 leader_election.cc:341] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.445086 1937 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.443991 1893 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:21.447604 1702 consensus_peers.cc:487] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Couldn't send request to peer 0fe98f486bab4d0b9c421c145d83f1ce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:21.456108 1893 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.458359 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:21.457592 1937 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.458945 1662 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.461549 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.462093 1735 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.453665 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.462643 1586 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.464648 1778 tablet_bootstrap.cc:654] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.466639 1735 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:21.450205 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.470541 2043 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.472108 1552 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.475440 1777 tablet_bootstrap.cc:654] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.476402 1937 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.477177 1937 raft_consensus.cc:686] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:30:21.481161 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 50912.2us
I20260927 23:30:21.481627 1779 tablet_bootstrap.cc:654] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.479417 1937 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.492770 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:21.494184 1663 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.509447 1662 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:21.510587 1778 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.511209 1778 ts_tablet_manager.cc:1397] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.010s
W20260927 23:30:21.521852 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:21.493477 1586 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:21.538082 1702 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.538923 1835 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.539338 1835 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:21.539716 1777 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.540144 1777 ts_tablet_manager.cc:1397] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.012s
I20260927 23:30:21.534437 1663 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:21.540135 1835 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.545811 1893 consensus_queue.cc:1035] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.551307 1777 raft_consensus.cc:348] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.552237 1777 raft_consensus.cc:374] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.552628 1777 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
W20260927 23:30:21.557677 1628 consensus_peers.cc:487] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Couldn't send request to peer 0fe98f486bab4d0b9c421c145d83f1ce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:21.558907 1778 raft_consensus.cc:348] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.560016 1778 raft_consensus.cc:374] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.560331 1778 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.561125 1778 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.556289 1777 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.574790 1778 ts_tablet_manager.cc:1428] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.063s user 0.006s sys 0.002s
I20260927 23:30:21.576337 1778 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.574644 1483 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:21.576383 2043 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 23:30:21.585853 1777 ts_tablet_manager.cc:1428] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.045s user 0.003s sys 0.005s
I20260927 23:30:21.586879 1777 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.583858 1937 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.588546 1937 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.591921 1937 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.602758 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.620160 1482 catalog_manager.cc:5495] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:21.625663 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:21.626904 1551 leader_election.cc:341] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:21.627532 1552 leader_election.cc:341] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.628021 1552 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.645764 2068 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.650522 1778 tablet_bootstrap.cc:654] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.652547 1734 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.667045 2043 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 23:30:21.680270 1781 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:21.685882 1779 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:21.686363 1779 ts_tablet_manager.cc:1397] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.250s user 0.009s sys 0.037s
I20260927 23:30:21.697738 1779 raft_consensus.cc:348] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.698724 1779 raft_consensus.cc:374] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.699016 1779 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.699791 1777 tablet_bootstrap.cc:654] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.699676 1779 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.704865 1779 ts_tablet_manager.cc:1428] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20260927 23:30:21.707036 1779 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.724478 1779 tablet_bootstrap.cc:654] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.748813 1778 tablet_bootstrap.cc:492] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.749264 1778 ts_tablet_manager.cc:1397] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.173s user 0.016s sys 0.014s
I20260927 23:30:21.751561 1778 raft_consensus.cc:348] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.752257 1778 raft_consensus.cc:374] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.752521 1778 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.753227 1778 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.755199 1778 ts_tablet_manager.cc:1428] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 23:30:21.760848 1779 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:21.761297 1779 ts_tablet_manager.cc:1397] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.007s
I20260927 23:30:21.764038 1779 raft_consensus.cc:348] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.764783 1779 raft_consensus.cc:374] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.765027 1779 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.765791 1779 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.766449 1777 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.767321 1777 ts_tablet_manager.cc:1397] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.181s user 0.013s sys 0.012s
I20260927 23:30:21.768122 1779 ts_tablet_manager.cc:1428] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 23:30:21.768600 1778 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.769026 1779 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.769775 1777 raft_consensus.cc:348] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.773684 1777 raft_consensus.cc:374] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.773988 1777 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.775946 1779 tablet_bootstrap.cc:654] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.777212 1778 tablet_bootstrap.cc:654] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.780125 1777 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.782238 1777 ts_tablet_manager.cc:1428] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260927 23:30:21.783346 1777 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.786746 1781 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.787456 1781 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.788614 1777 tablet_bootstrap.cc:654] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.790707 1893 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.791168 1893 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.792224 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.792869 1735 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:21.793200 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.793784 1587 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:21.794772 1781 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.795632 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.796491 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.797000 1854 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.797399 1587 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:21.798884 1893 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.797397 1854 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.800541 1854 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.808758 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.809834 1659 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:21.814518 2044 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.816610 2044 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.818604 1702 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.819898 1935 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.819977 1893 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.820580 1893 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.821089 1893 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.820808 1935 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.822903 1935 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.825248 1778 tablet_bootstrap.cc:492] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.825038 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.825714 1778 ts_tablet_manager.cc:1397] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.010s
W20260927 23:30:21.831611 1628 consensus_peers.cc:487] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Couldn't send request to peer 0fe98f486bab4d0b9c421c145d83f1ce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:21.830847 1854 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.830892 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.833981 1778 raft_consensus.cc:348] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.834628 1778 raft_consensus.cc:374] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.835078 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.835371 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.835296 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.836364 1778 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.833400 2044 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.836922 1587 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:21.836994 1628 leader_election.cc:341] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.836994 1778 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.838646 1935 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.838362 1854 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.839794 1778 ts_tablet_manager.cc:1428] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260927 23:30:21.840994 1778 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.844372 1587 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.845144 1778 tablet_bootstrap.cc:654] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.845602 1628 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:21.847575 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.848143 2044 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.848619 2044 raft_consensus.cc:686] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:21.849399 2044 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.849676 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.850847 1779 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
W20260927 23:30:21.850481 1627 leader_election.cc:341] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:21.851599 1779 ts_tablet_manager.cc:1397] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.016s
I20260927 23:30:21.851523 1627 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.852452 2041 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.853128 1777 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.853636 1777 ts_tablet_manager.cc:1397] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.070s user 0.028s sys 0.003s
I20260927 23:30:21.850629 1781 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260927 23:30:21.854192 1779 raft_consensus.cc:348] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.859004 1779 raft_consensus.cc:374] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.860504 1779 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.861078 1779 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.863233 1779 ts_tablet_manager.cc:1428] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.011s user 0.005s sys 0.002s
I20260927 23:30:21.864012 1779 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.850270 1735 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.864571 1777 raft_consensus.cc:348] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.870359 1777 raft_consensus.cc:374] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.870661 1777 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.874251 1777 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.877568 1779 tablet_bootstrap.cc:654] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.883414 1777 ts_tablet_manager.cc:1428] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20260927 23:30:21.884150 1735 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.886047 1893 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.889009 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:21.889621 1659 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.891495 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.892140 1588 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.894356 1893 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:21.901139 1588 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:21.902276 1702 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.903194 1893 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:21.903867 1893 raft_consensus.cc:686] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:21.904695 1893 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.906561 1659 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:21.909212 1777 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.912876 1482 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:21.932420 1485 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:21.936383 1777 tablet_bootstrap.cc:654] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.937774 1935 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.938285 1935 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.941339 1935 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.944063 1779 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:21.944590 1779 ts_tablet_manager.cc:1397] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.011s
I20260927 23:30:21.945219 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:21.949050 1778 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.949481 1778 ts_tablet_manager.cc:1397] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.109s user 0.020s sys 0.012s
I20260927 23:30:21.948246 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.951678 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:21.951872 1778 raft_consensus.cc:348] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.952510 1778 raft_consensus.cc:374] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.952253 1779 raft_consensus.cc:348] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.952826 1778 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.952870 1935 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:21.953150 1779 raft_consensus.cc:374] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.953732 1779 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.953514 1778 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.954372 1779 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.955564 2044 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:21.956022 2044 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.957899 2044 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.958688 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:21.959323 1588 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:21.959710 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:21.956605 1779 ts_tablet_manager.cc:1428] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260927 23:30:21.960659 1735 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:21.961243 1779 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:21.961714 1627 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:21.962527 1778 ts_tablet_manager.cc:1428] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 23:30:21.962815 2044 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:21.963210 2044 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:21.963351 1778 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:21.963578 2044 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.970556 1777 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:21.971565 1777 ts_tablet_manager.cc:1397] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.009s
I20260927 23:30:21.972317 1779 tablet_bootstrap.cc:654] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.972115 2044 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.974411 1777 raft_consensus.cc:348] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.975085 1777 raft_consensus.cc:374] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:21.975030 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:21.975517 1777 raft_consensus.cc:729] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:21.975832 1588 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.975890 2044 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:21.976115 1777 consensus_queue.cc:260] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:21.979969 1777 ts_tablet_manager.cc:1428] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:30:21.980258 1778 tablet_bootstrap.cc:654] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.983537 1588 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:21.989382 1777 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:21.994707 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:21.995316 1735 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:21.995896 1778 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:21.996328 1778 ts_tablet_manager.cc:1397] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.014s
I20260927 23:30:21.996490 1777 tablet_bootstrap.cc:654] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:21.999692 1778 raft_consensus.cc:348] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.000387 1778 raft_consensus.cc:374] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.000620 1778 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.001197 1778 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.002919 1778 ts_tablet_manager.cc:1428] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 23:30:22.005970 1778 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.014433 1628 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:22.018736 2044 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.019284 2044 raft_consensus.cc:686] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:22.020479 2044 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.022728 1778 tablet_bootstrap.cc:654] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.033828 1781 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.034409 1781 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.034600 1588 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.036717 1781 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.040930 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:22.042958 1627 leader_election.cc:341] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.044072 1777 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.044507 1777 ts_tablet_manager.cc:1397] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.022s
I20260927 23:30:22.044624 1786 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:22.037849 1735 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:22.042186 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:22.048166 1628 leader_election.cc:341] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.048568 1628 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.051122 1854 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:22.053822 1779 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.053673 1777 raft_consensus.cc:348] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.054301 1779 ts_tablet_manager.cc:1397] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.006s
I20260927 23:30:22.054381 1777 raft_consensus.cc:374] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.066166 1777 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.065023 1935 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 23:30:22.068629 1777 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.070204 1777 ts_tablet_manager.cc:1428] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260927 23:30:22.071142 1777 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.071352 1893 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:22.078225 1777 tablet_bootstrap.cc:654] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.087862 1779 raft_consensus.cc:348] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.088807 1779 raft_consensus.cc:374] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.089087 1779 raft_consensus.cc:729] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.091945 1779 consensus_queue.cc:260] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.096019 1777 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.096665 1777 ts_tablet_manager.cc:1397] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.002s
I20260927 23:30:22.097659 1779 ts_tablet_manager.cc:1428] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260927 23:30:22.100294 1779 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.106283 1777 raft_consensus.cc:348] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.107429 1777 raft_consensus.cc:374] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.107788 1777 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.104094 1893 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.111340 1893 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.113605 1893 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.117703 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:22.119068 1700 leader_election.cc:341] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.108902 1777 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.118673 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.163033 1778 tablet_bootstrap.cc:492] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.163015 1587 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:22.164249 1778 ts_tablet_manager.cc:1397] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.161s user 0.035s sys 0.027s
I20260927 23:30:22.164986 1779 tablet_bootstrap.cc:654] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.178283 1777 ts_tablet_manager.cc:1428] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.081s user 0.007s sys 0.003s
I20260927 23:30:22.177209 1482 catalog_manager.cc:5495] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:22.179271 1777 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.182034 1588 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.182160 1778 raft_consensus.cc:348] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.182921 1778 raft_consensus.cc:374] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.183167 1778 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.183830 1778 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.164754 1702 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:22.195042 1778 ts_tablet_manager.cc:1428] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20260927 23:30:22.197116 1778 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.200218 1893 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.200635 1893 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.200971 1893 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.206717 1835 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.207221 1835 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.209452 1835 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.230013 1586 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.230302 1778 tablet_bootstrap.cc:654] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.255005 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:22.259909 1660 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:22.264540 1700 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:22.266110 1835 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.266496 1835 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.266832 1835 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:22.254763 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 90375.5us
I20260927 23:30:22.282648 1777 tablet_bootstrap.cc:654] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.297974 1779 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.298545 1779 ts_tablet_manager.cc:1397] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.198s user 0.019s sys 0.003s
I20260927 23:30:22.303092 1779 raft_consensus.cc:348] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.304194 1779 raft_consensus.cc:374] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.304550 1779 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.308449 1893 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.300804 1835 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.313973 1835 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.305362 1779 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.318099 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:22.318753 1661 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.325870 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.302641 2099 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260927 23:30:22.336730 1779 ts_tablet_manager.cc:1428] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20260927 23:30:22.337651 1779 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.344475 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:22.338089 1893 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:22.350660 1700 leader_election.cc:341] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.353583 2099 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260927 23:30:22.370751 1778 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.371493 1778 ts_tablet_manager.cc:1397] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.176s user 0.014s sys 0.029s
I20260927 23:30:22.375288 1779 tablet_bootstrap.cc:654] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.378144 1661 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:22.381474 1700 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:22.382265 2099 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.383381 2099 raft_consensus.cc:686] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:22.386168 2099 consensus_queue.cc:237] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.414842 1853 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:22.425467 1778 raft_consensus.cc:348] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.426278 1778 raft_consensus.cc:374] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.427016 1777 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.427659 1777 ts_tablet_manager.cc:1397] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.249s user 0.020s sys 0.011s
I20260927 23:30:22.426570 1778 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.433103 1778 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.438367 1778 ts_tablet_manager.cc:1428] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.067s user 0.004s sys 0.005s
I20260927 23:30:22.438131 1777 raft_consensus.cc:348] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.438849 1777 raft_consensus.cc:374] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.439095 1777 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.439694 1777 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.437026 2043 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260927 23:30:22.447506 1777 ts_tablet_manager.cc:1428] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260927 23:30:22.452684 1777 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.460873 1778 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.462961 2043 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:22.468094 1483 catalog_manager.cc:5495] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
W20260927 23:30:22.480914 1702 leader_election.cc:341] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.491197 1778 tablet_bootstrap.cc:654] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.495230 1853 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.495818 1853 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.495282 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:22.497130 1779 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.497627 1779 ts_tablet_manager.cc:1397] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.160s user 0.030s sys 0.021s
I20260927 23:30:22.499061 1853 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:22.499444 1702 leader_election.cc:341] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.500394 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.505597 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:22.506296 2043 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:30:22.522500 1628 leader_election.cc:341] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.515553 1583 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.518771 1854 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.518894 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:22.535106 1779 raft_consensus.cc:348] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.541170 1779 raft_consensus.cc:374] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.519083 2099 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260927 23:30:22.541462 1779 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.542519 1777 tablet_bootstrap.cc:654] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.542112 1779 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.539858 1854 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.555814 1854 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.547150 1733 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:22.557753 1583 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:22.565061 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:22.566223 1628 leader_election.cc:341] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.578660 2100 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.579221 2100 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.583782 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:22.584400 1661 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:22.584573 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:22.585984 1552 leader_election.cc:341] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.580338 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:22.578612 1702 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:22.588449 1551 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.583105 2100 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.589095 1987 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.589419 1987 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.589735 1987 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.589861 2099 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.590421 2099 raft_consensus.cc:686] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:22.591178 2099 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.594336 1778 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.594810 1778 ts_tablet_manager.cc:1397] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.134s user 0.017s sys 0.009s
I20260927 23:30:22.598090 1779 ts_tablet_manager.cc:1428] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.100s user 0.006s sys 0.003s
I20260927 23:30:22.599212 1779 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.610035 1987 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.614243 1778 raft_consensus.cc:348] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.614661 1627 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:22.615021 1778 raft_consensus.cc:374] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.615315 1778 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
W20260927 23:30:22.615792 1627 leader_election.cc:341] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.615903 1778 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.616233 1627 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.616979 2043 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.617307 2043 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.617597 2043 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.617630 1781 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:22.618161 1778 ts_tablet_manager.cc:1428] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260927 23:30:22.619047 1778 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.620208 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:22.621445 1661 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.635491 1661 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:22.641891 1777 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.642387 1994 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.643718 1777 ts_tablet_manager.cc:1397] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.192s user 0.025s sys 0.006s
I20260927 23:30:22.639132 1987 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.646018 1551 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:22.647989 2103 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.648857 2103 raft_consensus.cc:686] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:22.654290 1777 raft_consensus.cc:348] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.650182 1994 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.655037 1777 raft_consensus.cc:374] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.655357 1777 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.655930 1777 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.656322 1994 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.658425 1777 ts_tablet_manager.cc:1428] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260927 23:30:22.658831 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:22.659308 1777 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.661253 1779 tablet_bootstrap.cc:654] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:22.662927 1552 leader_election.cc:341] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.663636 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:22.664278 1658 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:22.649583 2103 consensus_queue.cc:237] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.657382 2043 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.670217 1700 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:30:22.674302 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:22.672951 1482 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:22.673243 2043 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.688800 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.678198 1994 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.697597 1778 tablet_bootstrap.cc:654] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.687455 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:22.694752 1994 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.700254 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:22.705698 1737 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:22.707157 1628 leader_election.cc:341] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:22.706055 1702 leader_election.cc:341] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.704164 1994 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.715404 1779 tablet_bootstrap.cc:492] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.715880 1779 ts_tablet_manager.cc:1397] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.019s
I20260927 23:30:22.734673 1777 tablet_bootstrap.cc:654] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.741040 1994 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.741793 1779 raft_consensus.cc:348] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.743697 1779 raft_consensus.cc:374] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.743980 1779 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.744390 1994 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.746776 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:22.745707 1779 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.747442 1661 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.749944 1737 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:22.750262 1779 ts_tablet_manager.cc:1428] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260927 23:30:22.751168 1627 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:22.752640 1781 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.753120 1781 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:22.754046 1781 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.755574 1777 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.756031 1777 ts_tablet_manager.cc:1397] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.015s
I20260927 23:30:22.754087 1779 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.761236 1661 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:22.761476 1485 catalog_manager.cc:5495] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:22.762699 1700 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:22.764150 1777 raft_consensus.cc:348] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.764492 1893 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.765053 1893 raft_consensus.cc:686] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:22.765827 1893 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.767894 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:22.768985 1777 raft_consensus.cc:374] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.769541 1777 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.770339 1777 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.773137 1779 tablet_bootstrap.cc:654] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
W20260927 23:30:22.774369 1702 leader_election.cc:341] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.781013 1486 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:22.775367 1777 ts_tablet_manager.cc:1428] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260927 23:30:22.786172 1777 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.793370 1777 tablet_bootstrap.cc:654] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.796355 1778 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.796916 1778 ts_tablet_manager.cc:1397] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.178s user 0.015s sys 0.006s
I20260927 23:30:22.795445 1779 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.798987 1779 ts_tablet_manager.cc:1397] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.002s
I20260927 23:30:22.799320 1778 raft_consensus.cc:348] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.800009 1778 raft_consensus.cc:374] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.800264 1778 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.800285 1485 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:22.803599 1779 raft_consensus.cc:348] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.804194 1779 raft_consensus.cc:374] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.804430 1779 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.802487 1778 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.804961 1779 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.806891 1778 ts_tablet_manager.cc:1428] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 23:30:22.810422 1778 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.811018 1779 ts_tablet_manager.cc:1428] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260927 23:30:22.816817 1779 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.822747 1779 tablet_bootstrap.cc:654] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.838405 1893 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.838883 1893 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.840823 1893 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.841758 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:22.842121 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.842423 1661 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:22.842756 1587 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:22.843490 1700 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:22.844615 1893 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:22.845005 1893 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:22.845400 1893 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.847476 1777 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.847942 1777 ts_tablet_manager.cc:1397] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.017s
I20260927 23:30:22.848371 1779 tablet_bootstrap.cc:492] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.848810 1779 ts_tablet_manager.cc:1397] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.011s
I20260927 23:30:22.850435 1893 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.851132 1777 raft_consensus.cc:348] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.851845 1777 raft_consensus.cc:374] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.851519 1779 raft_consensus.cc:348] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.852252 1779 raft_consensus.cc:374] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.852239 1777 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.852629 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:22.853217 1587 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.853108 1777 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.855126 1777 ts_tablet_manager.cc:1428] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:30:22.856339 1777 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.856949 1893 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:22.857919 1778 tablet_bootstrap.cc:654] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.852633 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:22.858278 1661 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.862905 1777 tablet_bootstrap.cc:654] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.863797 2043 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.864238 2043 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.865895 1661 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:22.866161 2043 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.867326 1700 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:22.867534 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:22.868880 1628 leader_election.cc:341] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.869030 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:22.870242 1627 leader_election.cc:341] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:22.870687 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.870781 1587 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:22.871652 1893 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:22.872176 1893 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:22.852651 1779 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.873618 2043 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:22.872896 1893 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.874607 1777 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.874047 1779 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.875023 1777 ts_tablet_manager.cc:1397] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20260927 23:30:22.876612 1779 ts_tablet_manager.cc:1428] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260927 23:30:22.877496 1779 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:22.879168 1778 tablet_bootstrap.cc:492] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.879549 1778 ts_tablet_manager.cc:1397] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.000s
I20260927 23:30:22.878041 1777 raft_consensus.cc:348] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.880059 1777 raft_consensus.cc:374] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.880328 1777 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.881958 2043 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.881820 1777 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.882416 2043 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.885150 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:22.886076 1587 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 1: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:22.884145 1778 raft_consensus.cc:348] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.886739 1778 raft_consensus.cc:374] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.887884 1777 ts_tablet_manager.cc:1428] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260927 23:30:22.888247 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:22.889170 1777 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.889516 1779 tablet_bootstrap.cc:654] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.889585 1627 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.890370 1781 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.894979 1777 tablet_bootstrap.cc:654] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.896755 1781 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:22.887372 1778 raft_consensus.cc:729] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.897331 2043 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.897985 1778 consensus_queue.cc:260] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.900000 1778 ts_tablet_manager.cc:1428] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20260927 23:30:22.901275 1778 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.906935 1485 catalog_manager.cc:5495] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:22.911433 1778 tablet_bootstrap.cc:654] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.927034 1777 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:22.927623 1777 ts_tablet_manager.cc:1397] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.002s
I20260927 23:30:22.929885 1778 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:22.930312 1778 ts_tablet_manager.cc:1397] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.000s
I20260927 23:30:22.930601 1777 raft_consensus.cc:348] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.931584 1777 raft_consensus.cc:374] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.931886 1777 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.932485 1778 raft_consensus.cc:348] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.933233 1778 raft_consensus.cc:374] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.932644 1777 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.933494 1778 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.934255 1778 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.935824 1777 ts_tablet_manager.cc:1428] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:30:22.936035 1779 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:22.936417 1779 ts_tablet_manager.cc:1397] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.014s
I20260927 23:30:22.936542 1778 ts_tablet_manager.cc:1428] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 23:30:22.936676 1777 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:22.938102 1778 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:22.942919 1777 tablet_bootstrap.cc:654] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.945214 1778 tablet_bootstrap.cc:654] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:22.954365 1587 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.961302 1987 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:22.961774 1987 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.963614 1987 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:22.965046 1893 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:22.965956 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:22.966722 1658 raft_consensus.cc:2379] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 1: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:22.939327 1779 raft_consensus.cc:348] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:22.967154 1779 raft_consensus.cc:374] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:22.968565 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:22.970264 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:22.972316 2103 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:22.967500 1779 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:22.989341 1893 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:22.990483 2103 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:22.995807 1779 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.007133 2098 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:23.015094 2041 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.015584 2041 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.017462 2041 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.021055 1778 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:23.021495 1778 ts_tablet_manager.cc:1397] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.013s
I20260927 23:30:23.022126 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:23.025804 1628 leader_election.cc:341] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.025216 1778 raft_consensus.cc:348] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.026719 1778 raft_consensus.cc:374] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.026983 1778 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.027534 1778 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.029738 1778 ts_tablet_manager.cc:1428] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260927 23:30:23.031147 1778 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:23.033088 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:23.035343 1627 leader_election.cc:341] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.035768 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:23.036829 1778 tablet_bootstrap.cc:654] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.037451 2043 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.040572 1779 ts_tablet_manager.cc:1428] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.104s user 0.005s sys 0.002s
I20260927 23:30:23.054487 2099 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:23.055469 1778 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:23.058816 1778 ts_tablet_manager.cc:1397] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.012s
I20260927 23:30:23.056718 2103 consensus_queue.cc:1035] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:23.062695 1778 raft_consensus.cc:348] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.065721 1778 raft_consensus.cc:374] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.066275 1778 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.066972 1778 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.069018 1778 ts_tablet_manager.cc:1428] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260927 23:30:23.070183 1778 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:23.075655 1778 tablet_bootstrap.cc:654] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.079803 1779 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:23.081637 2103 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.082280 2103 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.084873 2103 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.086807 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.100638 1854 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.101770 1854 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.103484 1854 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.109804 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.104525 1658 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:23.112188 1551 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:23.113517 2101 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:23.113842 2101 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:23.114148 2101 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.129729 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.134430 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:23.138963 1627 leader_election.cc:341] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.098845 2098 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.148654 2098 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.150084 1737 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:23.151149 1854 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.151602 1854 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.154515 1854 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:23.162103 1628 leader_election.cc:341] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.162580 1628 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:23.175510 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.184182 1777 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:23.184660 1777 ts_tablet_manager.cc:1397] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.248s user 0.096s sys 0.118s
I20260927 23:30:23.184073 2098 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:23.167507 1781 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.180245 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.195061 1778 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:23.195941 1778 ts_tablet_manager.cc:1397] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.126s user 0.037s sys 0.052s
I20260927 23:30:23.191560 2101 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.234810 1778 raft_consensus.cc:348] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.235526 1778 raft_consensus.cc:374] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.235764 1778 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.236279 1778 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.236761 1779 tablet_bootstrap.cc:654] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.239305 2151 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.239959 2151 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.237684 1778 ts_tablet_manager.cc:1428] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.041s user 0.005s sys 0.001s
W20260927 23:30:23.242499 1627 leader_election.cc:341] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.243467 1778 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:23.245023 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.245833 1586 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:30:23.247413 1628 leader_election.cc:341] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.247828 1628 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:23.247620 1777 raft_consensus.cc:348] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.248474 1777 raft_consensus.cc:374] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.248956 1781 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.249437 2100 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.249647 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.249905 2100 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.250259 1659 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:23.251802 2100 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.248791 1777 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.254079 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.254732 1663 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:23.254874 1700 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:23.267369 1777 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.266500 1835 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:23.270766 1835 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:23.247601 2151 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.271173 1835 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.273557 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.273952 2101 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.272750 2098 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.275877 2098 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.278837 2098 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:23.280495 1777 ts_tablet_manager.cc:1428] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.096s user 0.004s sys 0.008s
I20260927 23:30:23.281359 1777 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Bootstrap starting.
I20260927 23:30:23.281665 1778 tablet_bootstrap.cc:654] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.282526 1551 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:23.285198 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.285823 1659 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:23.286249 2043 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:23.286660 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.289786 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.290771 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:23.290880 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.299012 2101 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:23.299507 2101 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:23.299906 2101 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.293371 1835 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.332850 1736 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 1: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:30:23.335237 1551 leader_election.cc:341] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:23.340883 1552 leader_election.cc:341] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.337677 1994 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.343612 1994 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.334802 2041 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.345206 2041 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.345532 1994 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:23.346582 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:23.347285 1663 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.347342 2041 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.355943 1552 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:23.356999 1552 leader_election.cc:341] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.360110 1663 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
W20260927 23:30:23.331593 1702 leader_election.cc:341] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.370756 1777 tablet_bootstrap.cc:654] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.374574 1700 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:23.375458 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.362015 2151 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.377858 1893 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:23.379626 1893 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:23.379951 1893 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:23.378434 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 46215.2us
I20260927 23:30:23.384315 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.385929 1778 tablet_bootstrap.cc:492] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
W20260927 23:30:23.386157 1628 leader_election.cc:341] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
W20260927 23:30:23.372340 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 76434.6us
I20260927 23:30:23.386438 1778 ts_tablet_manager.cc:1397] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.143s user 0.011s sys 0.009s
I20260927 23:30:23.383191 1736 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.390297 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.397522 1893 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.408969 1893 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:23.394132 1835 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:23.416560 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:23.418186 1587 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:23.421855 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:23.435375 1628 consensus_peers.cc:487] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Couldn't send request to peer 0fe98f486bab4d0b9c421c145d83f1ce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260927 23:30:23.435307 1627 leader_election.cc:341] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.436193 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:23.437418 1854 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.445917 1551 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:23.432066 2098 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260927 23:30:23.434634 2107 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260927 23:30:23.455644 1778 raft_consensus.cc:348] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.456327 1778 raft_consensus.cc:374] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.456365 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:23.456640 1778 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.456926 1661 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.457226 1778 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.450881 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:23.463020 2101 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.464066 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:23.464911 2101 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.466753 1660 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:23.453864 1987 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:23.490564 1987 raft_consensus.cc:686] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:23.491505 1987 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.500156 1779 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:23.518154 1779 ts_tablet_manager.cc:1397] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.474s user 0.019s sys 0.009s
I20260927 23:30:23.517620 1587 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:23.546912 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 92702.6us
I20260927 23:30:23.519762 1661 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:23.548620 1778 ts_tablet_manager.cc:1428] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.161s user 0.002s sys 0.004s
I20260927 23:30:23.549634 1777 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: No bootstrap required, opened a new log
I20260927 23:30:23.550064 1777 ts_tablet_manager.cc:1397] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Time spent bootstrapping tablet: real 0.269s user 0.013s sys 0.012s
I20260927 23:30:23.553040 1587 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:23.557217 1777 raft_consensus.cc:348] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.558034 1777 raft_consensus.cc:374] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.563697 1777 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.564476 1777 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.566752 1777 ts_tablet_manager.cc:1428] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260927 23:30:23.575068 1778 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:23.576655 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:23.577235 1663 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.587602 1663 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:23.589087 1551 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:23.573520 2155 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.592172 1987 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:23.592648 1987 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:23.593387 1987 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.594235 1779 raft_consensus.cc:348] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.595070 1779 raft_consensus.cc:374] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.595633 1779 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.596271 1779 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.607841 1779 ts_tablet_manager.cc:1428] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.089s user 0.005s sys 0.000s
I20260927 23:30:23.608558 1779 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:23.576655 1702 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:23.628908 1778 tablet_bootstrap.cc:654] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.639477 1700 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:23.644312 2158 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.601410 2155 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.650275 1700 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:30:23.647702 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75316.2us
I20260927 23:30:23.665583 1835 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:23.673002 1779 tablet_bootstrap.cc:654] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:23.680411 1835 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:23.692457 1835 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.681769 1781 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.754509 2168 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 23:30:23.744696 2155 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:23.742462 1781 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:23.788177 2163 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260927 23:30:23.795684 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:23.796644 1660 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
I20260927 23:30:23.797955 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:23.799057 1551 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:23.833447 2150 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:23.816442 2168 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260927 23:30:23.834062 2150 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:23.834856 2150 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:23.836300 1781 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:23.836786 2173 log.cc:924] Time spent T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260927 23:30:23.843000 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:23.846912 2161 consensus_queue.cc:1035] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260927 23:30:23.861425 1778 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:23.862095 1778 ts_tablet_manager.cc:1397] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.287s user 0.010s sys 0.009s
I20260927 23:30:23.876390 2168 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:23.887300 2171 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:23.887889 2171 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.895273 1779 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:23.895915 1779 ts_tablet_manager.cc:1397] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.287s user 0.028s sys 0.012s
W20260927 23:30:23.895263 1552 leader_election.cc:341] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.883526 2150 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.904888 2150 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:23.909597 1552 leader_election.cc:341] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:23.916290 1778 raft_consensus.cc:348] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:23.917335 1778 raft_consensus.cc:374] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:23.917665 1778 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:23.921205 1778 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:23.929773 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:23.947278 1485 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:24.020556 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:24.026134 2171 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:24.053550 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:24.054284 1580 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:24.056689 1729 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:24.065114 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:24.124176 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 162691us
I20260927 23:30:24.133193 1778 ts_tablet_manager.cc:1428] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.271s user 0.005s sys 0.011s
I20260927 23:30:24.139801 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:24.140609 1584 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:24.141919 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:24.143808 2185 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:24.148697 2185 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:24.149127 2185 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:24.151167 1552 leader_election.cc:341] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:24.152830 2188 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:30:24.156587 2177 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.147s user 0.003s sys 0.007s
I20260927 23:30:24.156764 1779 raft_consensus.cc:348] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.157493 1779 raft_consensus.cc:374] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.157752 1779 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.158044 1702 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:24.158390 1779 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.158867 2098 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:24.159217 2098 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:24.159597 2098 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.159929 1779 ts_tablet_manager.cc:1428] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.264s user 0.006s sys 0.005s
I20260927 23:30:24.160763 1779 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:24.163129 2185 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.168861 2185 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:24.173077 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:24.176105 1581 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:24.179350 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 54325.3us
I20260927 23:30:24.183285 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:24.184054 1659 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:24.187139 2098 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.207815 1778 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:24.202766 2098 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:24.226712 1779 tablet_bootstrap.cc:654] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.212486 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:24.239081 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:24.253630 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:24.254232 1658 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
W20260927 23:30:24.256664 1552 consensus_peers.cc:487] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Couldn't send request to peer a2a3317e8da742d9aefe4c251abd4566. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:24.254105 1728 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 1: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:24.277338 1581 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:24.316970 1658 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:24.319092 2162 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:24.321161 2162 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.327265 2162 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:24.331374 2172 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:24.331868 2172 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.334401 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:24.338310 1778 tablet_bootstrap.cc:654] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.339669 2149 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:24.341251 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:24.342300 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 77319.8us
I20260927 23:30:24.342410 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:24.343200 1660 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:24.342348 1663 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 0.
W20260927 23:30:24.349577 1552 leader_election.cc:341] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:24.356168 1551 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:24.357825 2103 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:24.358933 2103 raft_consensus.cc:686] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:24.360131 2103 consensus_queue.cc:237] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.360773 2172 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:24.361568 1700 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:24.363032 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:24.364215 2191 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:24.366744 2191 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:24.367923 2191 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.366330 2168 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 23:30:24.370901 1987 consensus_queue.cc:1035] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 23:30:24.373296 1779 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:24.373881 1779 ts_tablet_manager.cc:1397] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.213s user 0.009s sys 0.004s
I20260927 23:30:24.373793 1551 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:24.377393 1987 raft_consensus.cc:2781] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:24.378389 1987 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:24.378701 1987 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.381738 1779 raft_consensus.cc:348] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.382699 1779 raft_consensus.cc:374] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.382954 1779 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.386377 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:24.391645 1994 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:24.383636 1779 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:24.399658 1702 leader_election.cc:341] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:24.401356 1779 ts_tablet_manager.cc:1428] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260927 23:30:24.402480 1779 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:24.404006 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:24.404680 1659 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 0.
I20260927 23:30:24.409582 1779 tablet_bootstrap.cc:654] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.410494 1700 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:24.411550 1994 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:24.412395 1994 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:24.412727 1994 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.414582 1778 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:24.415241 1778 ts_tablet_manager.cc:1397] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.208s user 0.025s sys 0.036s
I20260927 23:30:24.418278 1778 raft_consensus.cc:348] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.419334 1778 raft_consensus.cc:374] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.419803 1778 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.420701 1778 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.429270 1778 ts_tablet_manager.cc:1428] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20260927 23:30:24.429999 1778 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:24.431913 1987 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.434823 1779 tablet_bootstrap.cc:492] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:24.435335 1779 ts_tablet_manager.cc:1397] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.013s
I20260927 23:30:24.435490 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:24.436105 1659 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.436285 1778 tablet_bootstrap.cc:654] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.439486 1779 raft_consensus.cc:348] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.440290 1779 raft_consensus.cc:374] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.440694 1779 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.441578 1779 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.443645 1779 ts_tablet_manager.cc:1428] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:30:24.445531 1779 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:24.450968 1994 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.453480 1994 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:24.454102 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:24.455372 1702 leader_election.cc:341] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:24.458381 1778 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:24.458993 1778 ts_tablet_manager.cc:1397] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.009s
I20260927 23:30:24.461993 1778 raft_consensus.cc:348] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.462657 1778 raft_consensus.cc:374] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.462921 1778 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.463586 2172 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:24.463457 1778 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.463991 2172 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.466809 2172 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:24.467509 1778 ts_tablet_manager.cc:1428] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260927 23:30:24.468408 1778 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Bootstrap starting.
I20260927 23:30:24.468729 2172 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:24.471881 1779 tablet_bootstrap.cc:654] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.474651 1778 tablet_bootstrap.cc:654] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.477499 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:24.492825 1482 catalog_manager.cc:5495] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:24.509261 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:24.509881 1736 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.513844 2165 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:24.514358 2165 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.516681 1736 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:24.517107 2165 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:24.517452 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:24.519660 1987 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:24.519786 1627 leader_election.cc:341] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:30:24.517733 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:24.520931 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:24.521484 1659 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:24.521960 2103 raft_consensus.cc:2781] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:24.521528 1658 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.530447 2103 raft_consensus.cc:686] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:24.531301 2103 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.545282 1485 catalog_manager.cc:5495] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:24.547694 1485 catalog_manager.cc:5495] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:24.563688 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:24.564457 1663 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 1: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:24.565413 1700 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:24.566507 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:24.566902 2168 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.567203 1584 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 0.
I20260927 23:30:24.584872 1778 tablet_bootstrap.cc:492] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: No bootstrap required, opened a new log
I20260927 23:30:24.582727 1658 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:24.633198 1778 ts_tablet_manager.cc:1397] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent bootstrapping tablet: real 0.165s user 0.048s sys 0.043s
I20260927 23:30:24.633504 1779 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:24.634570 1779 ts_tablet_manager.cc:1397] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.190s user 0.027s sys 0.050s
I20260927 23:30:24.634400 1700 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:24.638556 1628 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:24.638238 1778 raft_consensus.cc:348] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.640031 1778 raft_consensus.cc:374] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.640460 1778 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.648710 1854 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:30:24.649199 1854 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:24.649116 1835 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:24.649729 1854 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.650233 1835 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:24.650786 1584 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.651925 1835 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.657351 1778 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.671808 1778 ts_tablet_manager.cc:1428] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent starting tablet: real 0.038s user 0.006s sys 0.008s
I20260927 23:30:24.582605 2168 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:24.687595 1779 raft_consensus.cc:348] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.689265 1779 raft_consensus.cc:374] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.689633 1779 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.698194 1854 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.690564 1779 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.700865 1854 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:24.708371 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:24.709236 1736 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.717491 1779 ts_tablet_manager.cc:1428] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.082s user 0.007s sys 0.004s
I20260927 23:30:24.719245 1779 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Bootstrap starting.
I20260927 23:30:24.725092 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:24.731496 1779 tablet_bootstrap.cc:654] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Neither blocks nor log segments found. Creating new log.
I20260927 23:30:24.748433 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:24.758522 1779 tablet_bootstrap.cc:492] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: No bootstrap required, opened a new log
I20260927 23:30:24.759102 1779 ts_tablet_manager.cc:1397] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Time spent bootstrapping tablet: real 0.040s user 0.016s sys 0.005s
I20260927 23:30:24.762332 1779 raft_consensus.cc:348] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.763183 1779 raft_consensus.cc:374] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:30:24.763545 1779 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Initialized, Role: FOLLOWER
I20260927 23:30:24.764406 1779 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:24.768554 1779 ts_tablet_manager.cc:1428] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260927 23:30:24.772280 1581 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:24.786350 1987 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:24.787456 1736 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:24.793577 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:24.794672 2187 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:30:24.795329 2187 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:24.805806 1581 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:24.813804 2187 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:24.902099 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 111423us
I20260927 23:30:24.941097 2180 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:24.945892 2166 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:24.967278 2188 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 23:30:25.187165 2224 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.194s user 0.001s sys 0.000s
I20260927 23:30:25.194552 2182 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 23:30:25.202345 2101 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:25.220989 1485 catalog_manager.cc:5495] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:25.241178 2157 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:25.241711 2157 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:25.263374 2187 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:30:25.284477 2157 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:25.225565 2101 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:25.284132 2158 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
W20260927 23:30:25.317716 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56830.1us
W20260927 23:30:25.420683 1534 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 23:30:25.367902 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:25.485211 2101 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:25.502334 1580 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:25.517922 2165 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:25.514472 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:25.559864 1733 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:30:25.570601 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:25.618674 2166 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 23:30:25.661631 1702 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:25.662889 2171 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:25.665656 2172 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 23:30:25.667832 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65938.7us
I20260927 23:30:25.678434 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:25.724229 1482 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:25.726878 1552 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:25.744480 2151 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 23:30:25.775913 1485 catalog_manager.cc:5495] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:30:25.808740 396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:30:25.821318 2248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:25.842027 396 server_base.cc:1034] running on GCE node
W20260927 23:30:25.843633 2249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:30:25.848129 2251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:30:25.850857 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 23:30:25.851136 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 23:30:25.851317 396 hybrid_clock.cc:648] HybridClock initialized: now 1790551825851297 us; error 0 us; skew 500 ppm
I20260927 23:30:25.851830 396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:30:25.855012 396 webserver.cc:460] Webserver started at http://127.0.99.4:37217/ using document root <none> and password file <none>
I20260927 23:30:25.855512 396 fs_manager.cc:359] Metadata directory not provided
I20260927 23:30:25.855731 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:30:25.856084 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:30:25.857446 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-3-root/instance:
uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
format_stamp: "Formatted at 2026-09-27 23:30:25 on dist-test-slave-zkzt"
I20260927 23:30:25.863544 396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 23:30:25.868909 2256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:25.870093 396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 23:30:25.870469 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-3-root
uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
format_stamp: "Formatted at 2026-09-27 23:30:25 on dist-test-slave-zkzt"
I20260927 23:30:25.870764 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:30:25.885838 396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:30:25.887416 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:30:25.888872 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:30:25.892004 396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:30:25.892232 396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:25.892493 396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:30:25.892664 396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:30:25.977154 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:45509
I20260927 23:30:25.977317 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:45509 every 8 connection(s)
W20260927 23:30:25.985853 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800170fa0 after lost signal to thread 399
I20260927 23:30:25.986129 396 auto_leader_rebalancer-test.cc:413] add a tserver: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:30:25.987897 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08000bcce0 after lost signal to thread 1612
W20260927 23:30:25.988950 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002bbf80 after lost signal to thread 1687
W20260927 23:30:25.989705 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002464a0 after lost signal to thread 1761
I20260927 23:30:25.999109 2319 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38077
I20260927 23:30:25.999518 2319 heartbeater.cc:464] Registering TS with master...
I20260927 23:30:26.000190 2319 heartbeater.cc:511] Master 127.0.99.62:38077 requested a full tablet report, sending...
I20260927 23:30:26.002063 1483 ts_manager.cc:194] Registered new tserver with Master: 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:26.003558 1483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35356
I20260927 23:30:26.560226 1580 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:26.569545 1734 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.570688 1659 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.572420 2151 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:26.574276 2151 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:26.585642 2103 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.589007 1659 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.604389 1580 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:26.625445 1734 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.628566 1483 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:26.665617 1554 consensus_peers.cc:487] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e4c2a41313d4adcb6e037570a9d5882. This is attempt 1: this message will repeat every 5th retry.
W20260927 23:30:26.669945 1554 consensus_peers.cc:487] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58cf4a7fdcb146b0a364fa00e849e902. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:26.678072 1734 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.686913 1659 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.689623 2103 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:26.696535 2100 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:26.729374 2334 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.741161 1660 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.744099 1733 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.768204 1483 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:26.818887 1659 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:26.831673 1580 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.832044 1733 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.833209 2242 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:26.834817 2229 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 23:30:26.849206 1629 consensus_peers.cc:487] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bac3d0a2c8444f79b96925750c29b42. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:26.851054 2242 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.857043 1580 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.877683 1737 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.886863 1581 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:26.898270 1483 catalog_manager.cc:5495] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:26.922013 1554 consensus_peers.cc:487] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74c5e2f6966148308b5ccecb8d8f4313. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:26.925200 1737 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.927572 2103 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:26.929335 1659 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:26.931561 2334 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:26.944592 2100 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.945654 1659 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.972711 1482 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:26.971552 1733 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:26.981069 2348 ts_tablet_manager.cc:927] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:26.996229 2348 tablet_copy_client.cc:316] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:27.006384 2319 heartbeater.cc:503] Master 127.0.99.62:38077 was elected leader, sending a full tablet report...
I20260927 23:30:27.028402 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 6e4c2a41313d4adcb6e037570a9d5882 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:27.028959 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 6e4c2a41313d4adcb6e037570a9d5882 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-6e4c2a41313d4adcb6e037570a9d5882
I20260927 23:30:27.037921 1598 tablet_copy_source_session.cc:205] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:27.042455 2348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4c2a41313d4adcb6e037570a9d5882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:27.059352 2348 tablet_copy_client.cc:791] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:27.060091 2348 tablet_copy_client.cc:655] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:27.064294 2348 tablet_copy_client.cc:531] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:27.071977 2348 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:27.102640 2348 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:27.104353 2348 tablet_bootstrap.cc:492] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:27.105103 2348 ts_tablet_manager.cc:1397] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.016s
I20260927 23:30:27.107841 2348 raft_consensus.cc:348] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.108812 2348 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:27.109501 2348 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.116369 2352 ts_tablet_manager.cc:927] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:27.123786 2348 ts_tablet_manager.cc:1428] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20260927 23:30:27.124097 2352 tablet_copy_client.cc:316] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:27.130703 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 58cf4a7fdcb146b0a364fa00e849e902 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:27.131261 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 58cf4a7fdcb146b0a364fa00e849e902 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-58cf4a7fdcb146b0a364fa00e849e902
I20260927 23:30:27.133985 1597 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-6e4c2a41313d4adcb6e037570a9d5882 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:27.134402 1597 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-6e4c2a41313d4adcb6e037570a9d5882 on tablet 6e4c2a41313d4adcb6e037570a9d5882 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:27.142794 1598 tablet_copy_source_session.cc:205] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:27.146709 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58cf4a7fdcb146b0a364fa00e849e902. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:27.160470 2352 tablet_copy_client.cc:791] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:27.161154 2352 tablet_copy_client.cc:655] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:27.166420 2352 tablet_copy_client.cc:531] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:27.175793 2352 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:27.206157 2352 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:27.207007 2352 tablet_bootstrap.cc:492] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:27.207657 2352 ts_tablet_manager.cc:1397] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.020s
I20260927 23:30:27.209718 2352 raft_consensus.cc:348] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.210363 2352 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:27.210831 2352 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.213413 2352 ts_tablet_manager.cc:1428] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 23:30:27.215091 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-58cf4a7fdcb146b0a364fa00e849e902 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:27.215483 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-58cf4a7fdcb146b0a364fa00e849e902 on tablet 58cf4a7fdcb146b0a364fa00e849e902 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:27.389799 2352 ts_tablet_manager.cc:927] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:27.406605 2352 tablet_copy_client.cc:316] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:27.428368 2294 raft_consensus.cc:1203] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:27.441668 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 2bac3d0a2c8444f79b96925750c29b42 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:27.442286 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 2bac3d0a2c8444f79b96925750c29b42 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-2bac3d0a2c8444f79b96925750c29b42
I20260927 23:30:27.450384 1673 tablet_copy_source_session.cc:205] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:27.453986 2352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bac3d0a2c8444f79b96925750c29b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:27.464564 2352 tablet_copy_client.cc:791] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:27.465073 2352 tablet_copy_client.cc:655] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:27.468665 2352 tablet_copy_client.cc:531] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:27.475948 2352 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:27.510090 2352 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:27.519388 2352 tablet_bootstrap.cc:492] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:27.520085 2352 ts_tablet_manager.cc:1397] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.004s
I20260927 23:30:27.520691 2348 ts_tablet_manager.cc:927] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:27.522987 2348 tablet_copy_client.cc:316] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:27.522964 2352 raft_consensus.cc:348] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.524585 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 74c5e2f6966148308b5ccecb8d8f4313 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:27.524631 2352 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:27.525234 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 74c5e2f6966148308b5ccecb8d8f4313 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-74c5e2f6966148308b5ccecb8d8f4313
I20260927 23:30:27.525509 2352 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.530228 2352 ts_tablet_manager.cc:1428] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 23:30:27.532313 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2bac3d0a2c8444f79b96925750c29b42 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:27.532697 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2bac3d0a2c8444f79b96925750c29b42 on tablet 2bac3d0a2c8444f79b96925750c29b42 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:27.541306 1598 tablet_copy_source_session.cc:205] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:27.543926 2348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c5e2f6966148308b5ccecb8d8f4313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:27.558420 2348 tablet_copy_client.cc:791] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:27.558957 2348 tablet_copy_client.cc:655] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:27.563094 2348 tablet_copy_client.cc:531] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:27.573489 2348 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:27.596660 2348 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:27.597615 2348 tablet_bootstrap.cc:492] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:27.598079 2348 ts_tablet_manager.cc:1397] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.005s
I20260927 23:30:27.600577 2348 raft_consensus.cc:348] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.601125 2348 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:27.601732 2348 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:27.603829 2348 ts_tablet_manager.cc:1428] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 23:30:27.605470 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-74c5e2f6966148308b5ccecb8d8f4313 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:27.605852 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-74c5e2f6966148308b5ccecb8d8f4313 on tablet 74c5e2f6966148308b5ccecb8d8f4313 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:27.693840 2294 raft_consensus.cc:1203] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:27.788861 2294 raft_consensus.cc:1203] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260927 23:30:27.850476 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 55606.6us
I20260927 23:30:27.908265 2294 raft_consensus.cc:1203] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:28.003659 2103 raft_consensus.cc:1050] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:28.005909 2103 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.018165 2294 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.019774 2334 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.022886 1657 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.024184 2334 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:28.034538 1737 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.051504 2100 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:28.069540 2100 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.077742 2334 raft_consensus.cc:1050] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:28.080173 2334 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.083202 1737 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.091741 1659 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.092064 1483 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:28.093748 2294 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.139758 2293 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.141531 2372 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:28.144897 1657 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.148034 1737 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.149821 2372 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.156095 2372 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:28.185683 2100 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.192960 1657 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.195148 1733 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.210165 1580 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.222366 2294 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.223336 2293 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.224221 2100 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:28.224243 1483 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.231391 1733 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.233145 2372 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.234579 2294 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.244551 2103 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:28.249426 1733 raft_consensus.cc:2932] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.257625 2361 raft_consensus.cc:1050] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:28.260108 2361 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.273422 1485 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.285951 1581 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.296131 1657 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.297230 2294 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.297938 2334 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260927 23:30:28.300554 2367 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:28.315905 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4c2a41313d4adcb6e037570a9d5882 with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:28.316754 2103 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.321967 2294 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.323341 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6e4c2a41313d4adcb6e037570a9d5882 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:28.330045 2385 tablet_replica.cc:330] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:28.331933 2385 raft_consensus.cc:2229] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:28.332595 2385 raft_consensus.cc:2258] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:28.332123 1659 raft_consensus.cc:2932] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.338552 2385 ts_tablet_manager.cc:1905] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260927 23:30:28.349828 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 100257us
I20260927 23:30:28.359498 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 58cf4a7fdcb146b0a364fa00e849e902 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:28.363664 1581 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.365650 2294 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.367733 2363 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.367270 1485 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.371580 2365 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.393038 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cf4a7fdcb146b0a364fa00e849e902 with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:28.406188 2334 raft_consensus.cc:1050] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:28.408849 2334 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.415095 1733 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.427465 2365 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.444092 2294 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.444727 1737 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.449051 2367 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260927 23:30:28.452690 2103 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.456164 1656 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:28.457561 2103 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:28.458729 2385 ts_tablet_manager.cc:1918] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:28.459316 2385 log.cc:1196] T 6e4c2a41313d4adcb6e037570a9d5882 P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/6e4c2a41313d4adcb6e037570a9d5882
I20260927 23:30:28.469933 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 6e4c2a41313d4adcb6e037570a9d5882 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:28.476708 2389 tablet_replica.cc:330] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:28.487954 2361 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.505111 1581 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.527506 1733 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.527360 2293 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.535444 2103 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.539932 2292 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.558787 1485 catalog_manager.cc:5495] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.564759 2389 raft_consensus.cc:2229] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:28.574920 1733 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.577438 1659 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.575989 2389 raft_consensus.cc:2258] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:28.584838 2389 ts_tablet_manager.cc:1905] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:28.599985 1486 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.610960 2389 ts_tablet_manager.cc:1918] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:28.611462 2389 log.cc:1196] T 58cf4a7fdcb146b0a364fa00e849e902 P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/58cf4a7fdcb146b0a364fa00e849e902
I20260927 23:30:28.613893 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet 58cf4a7fdcb146b0a364fa00e849e902 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:28.620828 1656 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.626089 1733 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.626592 2293 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.627912 2395 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:28.630160 2361 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:28.635104 2363 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.638193 1579 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:28.638669 1733 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.645886 1659 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.648263 2383 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:28.646914 2293 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:28.659089 2383 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:28.661203 2372 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.664172 1657 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.665472 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2bac3d0a2c8444f79b96925750c29b42 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:28.670666 1483 catalog_manager.cc:5495] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.674437 2294 raft_consensus.cc:2932] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.674790 2292 raft_consensus.cc:2932] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:28.679195 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 74c5e2f6966148308b5ccecb8d8f4313 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:28.687765 1485 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:28.700580 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bac3d0a2c8444f79b96925750c29b42 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:28.710407 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c5e2f6966148308b5ccecb8d8f4313 with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:28.711380 2389 tablet_replica.cc:330] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:28.712304 2389 raft_consensus.cc:2229] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:28.712994 2389 raft_consensus.cc:2258] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:28.716156 2389 ts_tablet_manager.cc:1905] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:28.725044 2396 tablet_replica.cc:330] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:28.726599 2396 raft_consensus.cc:2229] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:28.727329 2396 raft_consensus.cc:2258] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:28.733116 2396 ts_tablet_manager.cc:1905] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:28.752892 2389 ts_tablet_manager.cc:1918] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:28.753311 2389 log.cc:1196] T 74c5e2f6966148308b5ccecb8d8f4313 P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/74c5e2f6966148308b5ccecb8d8f4313
I20260927 23:30:28.754557 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet 74c5e2f6966148308b5ccecb8d8f4313 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:28.755113 2396 ts_tablet_manager.cc:1918] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:28.755546 2396 log.cc:1196] T 2bac3d0a2c8444f79b96925750c29b42 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/2bac3d0a2c8444f79b96925750c29b42
I20260927 23:30:28.757212 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 2bac3d0a2c8444f79b96925750c29b42 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:29.801947 1733 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:29.810875 1586 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.812048 1657 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.812472 2398 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:29.815866 2398 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:29.836268 2381 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.842020 1586 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.860706 1660 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.856070 1733 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:29.867729 1485 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:29.902398 1703 consensus_peers.cc:487] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e382c82189e743b4ace885d63e5e9560. This is attempt 1: this message will repeat every 5th retry.
W20260927 23:30:29.937336 1703 consensus_peers.cc:487] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f797da3c81264653acefad127c4e4b2e. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:29.941228 1586 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.942772 1659 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.943006 2381 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:29.944273 2386 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:29.958778 2386 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.962564 1580 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.962564 1659 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:29.974015 1660 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:29.981786 1486 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:29.993820 1629 consensus_peers.cc:487] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1101ef7672c345ec8f5509edeb86b553. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:29.994076 1733 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.998472 1580 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:29.999476 2395 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:30.005402 2363 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:30.021144 2395 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.022646 1734 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.025831 1580 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.041203 1485 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.060026 1734 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
W20260927 23:30:30.070574 1703 consensus_peers.cc:487] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2686f0078ab94fbd99b4744ba43a0980. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:30.070752 1659 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:30.071871 1580 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:30.073202 2386 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:30.075484 2406 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:30.085665 2386 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.089111 1659 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.093955 1580 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.110388 1486 catalog_manager.cc:5495] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:30.300338 2421 ts_tablet_manager.cc:927] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:30.302556 2421 tablet_copy_client.cc:316] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:30.329519 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet e382c82189e743b4ace885d63e5e9560 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:30.330072 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet e382c82189e743b4ace885d63e5e9560 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-e382c82189e743b4ace885d63e5e9560
I20260927 23:30:30.338989 1747 tablet_copy_source_session.cc:205] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:30.341815 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e382c82189e743b4ace885d63e5e9560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:30.356436 2421 tablet_copy_client.cc:791] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:30.357028 2421 tablet_copy_client.cc:655] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:30.361544 2421 tablet_copy_client.cc:531] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:30.371165 2421 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:30.392745 2421 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:30.393570 2421 tablet_bootstrap.cc:492] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:30.394181 2421 ts_tablet_manager.cc:1397] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20260927 23:30:30.396804 2421 raft_consensus.cc:348] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.397617 2421 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:30.398202 2421 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.404966 2421 ts_tablet_manager.cc:1428] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 23:30:30.406585 1747 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e382c82189e743b4ace885d63e5e9560 received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:30.406960 1747 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e382c82189e743b4ace885d63e5e9560 on tablet e382c82189e743b4ace885d63e5e9560 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:30.523154 2421 ts_tablet_manager.cc:927] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:30.524945 2421 tablet_copy_client.cc:316] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:30.526713 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 1101ef7672c345ec8f5509edeb86b553 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:30.527289 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 1101ef7672c345ec8f5509edeb86b553 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-1101ef7672c345ec8f5509edeb86b553
I20260927 23:30:30.561724 2429 ts_tablet_manager.cc:927] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:30.564018 2428 ts_tablet_manager.cc:927] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:30.566195 2429 tablet_copy_client.cc:316] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:30.566910 1673 tablet_copy_source_session.cc:205] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:30.567711 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet 2686f0078ab94fbd99b4744ba43a0980 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:30.568222 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet 2686f0078ab94fbd99b4744ba43a0980 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-2686f0078ab94fbd99b4744ba43a0980
I20260927 23:30:30.568833 2428 tablet_copy_client.cc:316] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:30.570375 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1101ef7672c345ec8f5509edeb86b553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:30.570245 1746 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet f797da3c81264653acefad127c4e4b2e from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:30.570899 1746 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet f797da3c81264653acefad127c4e4b2e from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-f797da3c81264653acefad127c4e4b2e
I20260927 23:30:30.578282 1746 tablet_copy_source_session.cc:205] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:30.583698 2428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797da3c81264653acefad127c4e4b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:30.586714 1747 tablet_copy_source_session.cc:205] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:30.590915 2429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2686f0078ab94fbd99b4744ba43a0980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:30.606032 2421 tablet_copy_client.cc:791] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:30.606622 2421 tablet_copy_client.cc:655] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:30.611783 2421 tablet_copy_client.cc:531] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:30.622454 2429 tablet_copy_client.cc:791] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:30.623003 2429 tablet_copy_client.cc:655] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:30.623399 2428 tablet_copy_client.cc:791] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:30.623925 2428 tablet_copy_client.cc:655] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:30.631968 2428 tablet_copy_client.cc:531] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:30.633821 2429 tablet_copy_client.cc:531] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:30.635855 2421 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:30.653152 2429 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:30.661093 2428 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:30.672560 2421 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:30.673439 2421 tablet_bootstrap.cc:492] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:30.674121 2421 ts_tablet_manager.cc:1397] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.014s
I20260927 23:30:30.677101 2421 raft_consensus.cc:348] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.678048 2421 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:30.678897 2421 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.688500 2421 ts_tablet_manager.cc:1428] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260927 23:30:30.704260 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-1101ef7672c345ec8f5509edeb86b553 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:30.705436 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-1101ef7672c345ec8f5509edeb86b553 on tablet 1101ef7672c345ec8f5509edeb86b553 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:30.715727 2428 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:30.722637 2429 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:30.723590 2429 tablet_bootstrap.cc:492] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:30.724285 2429 ts_tablet_manager.cc:1397] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.071s user 0.024s sys 0.008s
I20260927 23:30:30.716660 2428 tablet_bootstrap.cc:492] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:30.727051 2428 ts_tablet_manager.cc:1397] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.066s user 0.016s sys 0.011s
I20260927 23:30:30.729028 2429 raft_consensus.cc:348] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.730047 2429 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:30.729801 2428 raft_consensus.cc:348] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.730775 2428 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:30.730882 2429 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.731472 2428 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:30.736301 2429 ts_tablet_manager.cc:1428] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260927 23:30:30.737790 1747 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2686f0078ab94fbd99b4744ba43a0980 received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:30.738140 1747 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2686f0078ab94fbd99b4744ba43a0980 on tablet 2686f0078ab94fbd99b4744ba43a0980 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:30.744576 2428 ts_tablet_manager.cc:1428] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260927 23:30:30.746229 1747 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f797da3c81264653acefad127c4e4b2e received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:30.746649 1747 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f797da3c81264653acefad127c4e4b2e on tablet f797da3c81264653acefad127c4e4b2e with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:30.819219 2294 raft_consensus.cc:1203] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:31.007097 2294 raft_consensus.cc:1203] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:31.045533 2294 raft_consensus.cc:1203] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:31.051307 2294 raft_consensus.cc:1203] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:31.391325 2395 raft_consensus.cc:1050] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:31.393529 2395 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.399806 2294 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.401144 1586 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.401676 2386 raft_consensus.cc:1050] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:31.401191 2395 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:31.404064 2386 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.405004 2363 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.407552 1737 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.410413 2395 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:31.433135 2395 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.435788 2292 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.437253 2381 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.440217 1737 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.443696 1586 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.444255 1659 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.447907 2406 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.450991 2381 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.455813 1586 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.461272 2406 raft_consensus.cc:1050] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:31.457742 1485 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:31.439591 2294 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.463552 2406 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.489547 2386 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.499208 2294 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.508579 1657 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.514722 1656 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.514727 2293 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.520358 2447 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.521935 2381 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.533967 2406 raft_consensus.cc:1050] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:31.535702 1586 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.537879 1580 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.534056 1483 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:31.547281 1656 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.556553 2293 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.558815 2447 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.557928 1580 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.559368 2395 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.560683 2435 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.564414 2406 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.568591 2435 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.572598 2294 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.581809 1580 raft_consensus.cc:2932] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.587759 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1101ef7672c345ec8f5509edeb86b553 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:31.599524 1485 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.609310 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 1101ef7672c345ec8f5509edeb86b553 with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:31.611863 1657 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.608763 2386 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.622448 1579 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.628197 2456 tablet_replica.cc:330] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:31.628058 2294 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.629179 2456 raft_consensus.cc:2229] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:31.629806 2456 raft_consensus.cc:2258] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:31.637238 1486 catalog_manager.cc:5495] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.644383 2456 ts_tablet_manager.cc:1905] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:31.650147 2293 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.652123 1657 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.653439 2406 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.654376 2386 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.661847 1586 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:31.665148 2386 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:31.673583 1737 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.674117 1734 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.680670 2294 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.680840 1660 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.682262 2292 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.688151 1581 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.688974 2447 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.689759 2381 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:31.691798 2381 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.692045 2445 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:31.711778 2386 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.713981 1656 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.716398 2406 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.719307 2292 raft_consensus.cc:2932] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.721365 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2686f0078ab94fbd99b4744ba43a0980 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:31.724331 2447 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.725652 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e382c82189e743b4ace885d63e5e9560 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:31.726878 1581 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.728894 1485 catalog_manager.cc:5495] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:31.731446 2293 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.751700 1482 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.769052 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 2686f0078ab94fbd99b4744ba43a0980 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:31.776619 1656 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.782152 1579 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.767338 2293 raft_consensus.cc:2932] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.771482 2456 ts_tablet_manager.cc:1918] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:31.801199 2456 log.cc:1196] T 1101ef7672c345ec8f5509edeb86b553 P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/1101ef7672c345ec8f5509edeb86b553
I20260927 23:30:31.805159 1485 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.810787 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet 1101ef7672c345ec8f5509edeb86b553 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:31.811460 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet e382c82189e743b4ace885d63e5e9560 with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:31.812428 2461 tablet_replica.cc:330] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:31.823149 2461 raft_consensus.cc:2229] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:31.823740 2461 raft_consensus.cc:2258] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:31.843369 2461 ts_tablet_manager.cc:1905] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:31.849558 2462 tablet_replica.cc:330] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:31.850618 2462 raft_consensus.cc:2229] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:31.851208 2462 raft_consensus.cc:2258] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:31.854372 2462 ts_tablet_manager.cc:1905] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:31.865752 2461 ts_tablet_manager.cc:1918] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:31.867086 2461 log.cc:1196] T 2686f0078ab94fbd99b4744ba43a0980 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/2686f0078ab94fbd99b4744ba43a0980
I20260927 23:30:31.868188 1733 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:31.869275 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 2686f0078ab94fbd99b4744ba43a0980 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:31.876410 2293 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.877498 1656 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:31.878665 2447 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.880223 2447 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:31.888996 2406 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.890847 1656 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.896332 2292 raft_consensus.cc:2932] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:31.900138 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f797da3c81264653acefad127c4e4b2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:31.902449 2462 ts_tablet_manager.cc:1918] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:31.902931 2462 log.cc:1196] T e382c82189e743b4ace885d63e5e9560 P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/e382c82189e743b4ace885d63e5e9560
I20260927 23:30:31.904646 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet e382c82189e743b4ace885d63e5e9560 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:31.907558 1482 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:31.920046 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet f797da3c81264653acefad127c4e4b2e with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:31.924288 2461 tablet_replica.cc:330] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:31.925274 2461 raft_consensus.cc:2229] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:31.925943 2461 raft_consensus.cc:2258] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:31.929211 2461 ts_tablet_manager.cc:1905] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:31.945571 2461 ts_tablet_manager.cc:1918] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:31.945926 2461 log.cc:1196] T f797da3c81264653acefad127c4e4b2e P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/f797da3c81264653acefad127c4e4b2e
I20260927 23:30:31.948064 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet f797da3c81264653acefad127c4e4b2e (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:32.969964 1733 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:32.976070 1579 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:32.976070 1660 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:32.977155 1703 consensus_peers.cc:487] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99c7c13b00be47d38145b16ad8a788cc. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:32.977449 2464 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:32.978701 2447 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:32.993685 2464 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:32.995204 1579 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:32.996320 1656 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.014158 1483 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.020480 1660 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.039196 1733 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:33.042173 2435 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260927 23:30:33.046721 1629 consensus_peers.cc:487] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3565b68068904064a965cd0a82a20802. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:33.045981 1579 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:33.048894 2463 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:33.058279 2463 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.065110 1733 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.066247 1579 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.082052 1482 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.083485 1660 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.091549 1733 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:33.091821 1629 consensus_peers.cc:487] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa9f6499247b4a1e9ef1440ac8906b50. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:33.092437 1579 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:33.093108 2435 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:33.094076 2463 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:33.100869 2463 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.102126 1733 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.110368 1579 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.116142 1581 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.129580 1483 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:33.143477 1554 consensus_peers.cc:487] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fb9ba54d39e48e08eefe27bddb5fd3b. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:33.146229 1660 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:33.147013 1734 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:33.148897 2471 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:33.153282 2470 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:33.156724 2471 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.164827 1733 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.183395 1482 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:33.191303 1660 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:33.364871 2484 ts_tablet_manager.cc:927] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:33.366808 2484 tablet_copy_client.cc:316] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:33.368214 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet 99c7c13b00be47d38145b16ad8a788cc from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:33.368729 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet 99c7c13b00be47d38145b16ad8a788cc from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-99c7c13b00be47d38145b16ad8a788cc
I20260927 23:30:33.374809 1747 tablet_copy_source_session.cc:205] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:33.377974 2484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c7c13b00be47d38145b16ad8a788cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:33.391793 2484 tablet_copy_client.cc:791] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:33.392305 2484 tablet_copy_client.cc:655] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:33.396281 2484 tablet_copy_client.cc:531] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:33.403656 2484 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:33.423452 2484 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:33.424660 2484 tablet_bootstrap.cc:492] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:33.425303 2484 ts_tablet_manager.cc:1397] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20260927 23:30:33.427912 2484 raft_consensus.cc:348] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.428737 2484 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:33.429328 2484 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.432721 2484 ts_tablet_manager.cc:1428] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 23:30:33.434526 1747 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-99c7c13b00be47d38145b16ad8a788cc received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:33.434886 1747 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-99c7c13b00be47d38145b16ad8a788cc on tablet 99c7c13b00be47d38145b16ad8a788cc with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:33.529402 2484 ts_tablet_manager.cc:927] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:33.538976 2484 tablet_copy_client.cc:316] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:33.540405 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:33.540879 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 2fb9ba54d39e48e08eefe27bddb5fd3b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-2fb9ba54d39e48e08eefe27bddb5fd3b
I20260927 23:30:33.547787 1598 tablet_copy_source_session.cc:205] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:33.550928 2484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:33.564781 2484 tablet_copy_client.cc:791] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:33.565318 2484 tablet_copy_client.cc:655] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:33.568473 2484 tablet_copy_client.cc:531] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:33.576334 2484 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:33.594220 2484 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:33.595014 2484 tablet_bootstrap.cc:492] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:33.595542 2484 ts_tablet_manager.cc:1397] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20260927 23:30:33.598042 2484 raft_consensus.cc:348] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.598685 2484 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:33.599188 2484 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.601806 2484 ts_tablet_manager.cc:1428] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 23:30:33.603638 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2fb9ba54d39e48e08eefe27bddb5fd3b received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:33.604018 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2fb9ba54d39e48e08eefe27bddb5fd3b on tablet 2fb9ba54d39e48e08eefe27bddb5fd3b with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:33.632230 2484 ts_tablet_manager.cc:927] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:33.634351 2484 tablet_copy_client.cc:316] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:33.636808 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 3565b68068904064a965cd0a82a20802 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:33.637200 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 3565b68068904064a965cd0a82a20802 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-3565b68068904064a965cd0a82a20802
I20260927 23:30:33.645788 1673 tablet_copy_source_session.cc:205] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:33.649667 2484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565b68068904064a965cd0a82a20802. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:33.670478 2484 tablet_copy_client.cc:791] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:33.671063 2484 tablet_copy_client.cc:655] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:33.673028 2491 ts_tablet_manager.cc:927] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:33.677773 2484 tablet_copy_client.cc:531] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:33.695654 2484 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:33.705418 2491 tablet_copy_client.cc:316] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:33.708647 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet fa9f6499247b4a1e9ef1440ac8906b50 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:33.709237 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet fa9f6499247b4a1e9ef1440ac8906b50 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-fa9f6499247b4a1e9ef1440ac8906b50
I20260927 23:30:33.716465 1673 tablet_copy_source_session.cc:205] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:33.719115 2491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa9f6499247b4a1e9ef1440ac8906b50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:33.731119 2484 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:33.731905 2484 tablet_bootstrap.cc:492] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:33.732481 2484 ts_tablet_manager.cc:1397] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.008s
I20260927 23:30:33.734980 2491 tablet_copy_client.cc:791] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:33.735518 2491 tablet_copy_client.cc:655] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:33.735656 2484 raft_consensus.cc:348] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.736465 2484 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:33.737040 2484 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.738780 2491 tablet_copy_client.cc:531] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:33.740149 2484 ts_tablet_manager.cc:1428] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260927 23:30:33.741760 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-3565b68068904064a965cd0a82a20802 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:33.742159 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-3565b68068904064a965cd0a82a20802 on tablet 3565b68068904064a965cd0a82a20802 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:33.747539 2491 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:33.766821 2491 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:33.767657 2491 tablet_bootstrap.cc:492] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:33.768263 2491 ts_tablet_manager.cc:1397] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260927 23:30:33.770761 2491 raft_consensus.cc:348] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.771566 2491 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:33.772234 2491 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:33.774585 2491 ts_tablet_manager.cc:1428] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 23:30:33.776513 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-fa9f6499247b4a1e9ef1440ac8906b50 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:33.776904 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-fa9f6499247b4a1e9ef1440ac8906b50 on tablet fa9f6499247b4a1e9ef1440ac8906b50 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:33.863931 2293 raft_consensus.cc:1203] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:34.068343 2293 raft_consensus.cc:1203] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:34.125121 2293 raft_consensus.cc:1203] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:34.218865 2293 raft_consensus.cc:1203] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:34.383496 2495 raft_consensus.cc:1050] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:34.385586 2495 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.392035 2293 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.392367 1586 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.392382 1657 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.394415 2447 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.395169 2495 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.396982 2498 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.407719 2447 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.409109 2292 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.426088 1482 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:34.434628 1657 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.437081 1581 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.496371 1737 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.501168 2292 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.501317 1586 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.502406 2498 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:34.504190 2498 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:34.510802 2495 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.512573 2292 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.517287 2481 raft_consensus.cc:1050] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:34.516954 1581 raft_consensus.cc:2932] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.520015 2481 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.522691 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 99c7c13b00be47d38145b16ad8a788cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:34.533752 1485 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.539793 2397 raft_consensus.cc:1050] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:34.542174 2397 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.555536 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c7c13b00be47d38145b16ad8a788cc with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:34.557669 1660 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.559018 1734 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.559284 2383 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.560557 2509 tablet_replica.cc:330] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:34.561575 2509 raft_consensus.cc:2229] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:34.562183 2509 raft_consensus.cc:2258] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:34.556764 2292 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.566097 2397 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:34.566485 2509 ts_tablet_manager.cc:1905] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:34.583516 2509 ts_tablet_manager.cc:1918] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:34.584168 2509 log.cc:1196] T 99c7c13b00be47d38145b16ad8a788cc P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/99c7c13b00be47d38145b16ad8a788cc
I20260927 23:30:34.586115 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 99c7c13b00be47d38145b16ad8a788cc (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:34.585865 2397 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260927 23:30:34.590364 2383 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.599277 2292 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.599342 2293 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.602984 1737 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.604854 1586 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.606429 2496 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.607609 1660 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.613319 2435 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.622377 1485 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:34.622444 2481 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:34.634091 2435 raft_consensus.cc:1050] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:34.638964 2435 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.669590 2293 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.671119 2516 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:34.672812 1737 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.674494 2293 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.676626 2496 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:34.686167 1733 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.673044 2513 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.713768 1580 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.715255 1579 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:34.724638 2292 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.726112 2383 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:34.732972 1483 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.736065 2521 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.737676 2292 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.748934 1734 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.766883 1483 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.773710 1730 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.774777 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:34.779946 1657 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:34.792277 2521 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:34.792928 2292 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.794286 2518 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:34.800981 2481 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.802261 1734 raft_consensus.cc:2932] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.808766 1737 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:34.809190 2291 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.810202 2435 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:34.813446 1737 raft_consensus.cc:2932] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.775831 2513 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260927 23:30:34.819787 2516 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.824961 1734 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.832999 2291 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.870659 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3565b68068904064a965cd0a82a20802 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:34.877665 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb9ba54d39e48e08eefe27bddb5fd3b with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:34.834612 1581 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.887341 1486 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:34.903772 2509 tablet_replica.cc:330] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:34.989933 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 3565b68068904064a965cd0a82a20802 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:34.992735 1485 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:35.009362 2528 tablet_replica.cc:330] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:35.012998 2528 raft_consensus.cc:2229] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:35.014578 2528 pending_rounds.cc:62] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Trying to abort 1 pending ops.
I20260927 23:30:35.015147 2509 raft_consensus.cc:2229] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:35.016487 2509 raft_consensus.cc:2258] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:35.014911 2528 pending_rounds.cc:68] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7334100314187636736 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3565b68068904064a965cd0a82a20802" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } } }
I20260927 23:30:35.018129 2528 raft_consensus.cc:2894] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260927 23:30:35.018740 2528 raft_consensus.cc:2866] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260927 23:30:35.019143 2528 raft_consensus.cc:2258] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:35.023847 2528 ts_tablet_manager.cc:1905] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:35.035538 2509 ts_tablet_manager.cc:1905] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:35.067507 2509 ts_tablet_manager.cc:1918] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:35.068131 2509 log.cc:1196] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/2fb9ba54d39e48e08eefe27bddb5fd3b
I20260927 23:30:35.071447 2528 ts_tablet_manager.cc:1918] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:35.071565 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 2fb9ba54d39e48e08eefe27bddb5fd3b (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:35.072031 2528 log.cc:1196] T 3565b68068904064a965cd0a82a20802 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/3565b68068904064a965cd0a82a20802
I20260927 23:30:35.074523 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 3565b68068904064a965cd0a82a20802 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:35.363628 1656 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:35.367939 2291 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:35.368136 1579 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:35.369254 2518 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:35.370172 2513 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:35.379438 2513 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:35.384948 2293 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:35.389302 1586 raft_consensus.cc:2932] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:35.393282 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fa9f6499247b4a1e9ef1440ac8906b50 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:35.408439 1483 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:35.417716 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet fa9f6499247b4a1e9ef1440ac8906b50 with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:35.421828 2532 tablet_replica.cc:330] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:35.422640 2532 raft_consensus.cc:2229] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:35.423254 2532 raft_consensus.cc:2258] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:35.425513 2532 ts_tablet_manager.cc:1905] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:35.439034 2532 ts_tablet_manager.cc:1918] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:35.439373 2532 log.cc:1196] T fa9f6499247b4a1e9ef1440ac8906b50 P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/fa9f6499247b4a1e9ef1440ac8906b50
I20260927 23:30:35.440907 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet fa9f6499247b4a1e9ef1440ac8906b50 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:36.478116 1657 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:36.483995 1580 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:36.484145 1629 consensus_peers.cc:487] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0bbde2432f547cbab3588c16fb623cf. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:36.484526 1734 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.485946 2435 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:36.487200 2534 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:36.495963 2534 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.503234 1580 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.516676 1734 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.519487 1580 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:36.520186 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:36.534853 1554 consensus_peers.cc:487] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6bcce456c08487491b881518f4f8b46. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:36.536433 1657 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.536932 1733 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.538358 2383 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:36.539310 2397 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:36.551450 2397 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.553252 1733 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.565763 1657 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.573570 1733 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:36.575109 1484 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:36.595057 1703 consensus_peers.cc:487] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e412df23ce8e4e71bc3c5bc171ca0e85. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:36.597627 1656 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.603003 1580 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.604744 2530 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:36.609126 2530 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:36.618839 2529 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.620843 1657 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.625627 1580 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.647449 1586 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:36.649691 1484 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:36.667449 1657 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.669088 2546 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 23:30:36.667429 1554 consensus_peers.cc:487] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f575009df7d846d997a605fb1e38ad31. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:36.675585 1733 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:36.680153 2546 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:36.682991 2383 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.694482 1657 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.696234 1733 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:36.706568 1483 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:37.021751 2553 ts_tablet_manager.cc:927] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:37.023783 2553 tablet_copy_client.cc:316] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:37.025772 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet d0bbde2432f547cbab3588c16fb623cf from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:37.026376 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet d0bbde2432f547cbab3588c16fb623cf from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-d0bbde2432f547cbab3588c16fb623cf
I20260927 23:30:37.034726 1673 tablet_copy_source_session.cc:205] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:37.038059 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bbde2432f547cbab3588c16fb623cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:37.051918 2556 ts_tablet_manager.cc:927] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:37.056077 2556 tablet_copy_client.cc:316] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:37.058059 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet f575009df7d846d997a605fb1e38ad31 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:37.059348 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet f575009df7d846d997a605fb1e38ad31 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-f575009df7d846d997a605fb1e38ad31
I20260927 23:30:37.061405 2555 ts_tablet_manager.cc:927] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:37.063858 2553 tablet_copy_client.cc:791] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:37.064653 2553 tablet_copy_client.cc:655] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:37.069499 2553 tablet_copy_client.cc:531] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:37.069844 2555 tablet_copy_client.cc:316] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:37.072758 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet e412df23ce8e4e71bc3c5bc171ca0e85 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:37.073378 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet e412df23ce8e4e71bc3c5bc171ca0e85 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-e412df23ce8e4e71bc3c5bc171ca0e85
I20260927 23:30:37.076263 1598 tablet_copy_source_session.cc:205] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:37.079245 2553 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:37.079591 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575009df7d846d997a605fb1e38ad31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:37.081630 1747 tablet_copy_source_session.cc:205] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:37.087137 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e412df23ce8e4e71bc3c5bc171ca0e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:37.097044 2556 tablet_copy_client.cc:791] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:37.097761 2556 tablet_copy_client.cc:655] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:37.101747 2556 tablet_copy_client.cc:531] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:37.105185 2559 ts_tablet_manager.cc:927] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:37.113773 2553 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:37.116185 2553 tablet_bootstrap.cc:492] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:37.116840 2553 ts_tablet_manager.cc:1397] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.008s
I20260927 23:30:37.120621 2556 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:37.121852 2553 raft_consensus.cc:348] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.122758 2553 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:37.123327 2553 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.126372 2555 tablet_copy_client.cc:791] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:37.127017 2555 tablet_copy_client.cc:655] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:37.133037 2553 ts_tablet_manager.cc:1428] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260927 23:30:37.137720 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d0bbde2432f547cbab3588c16fb623cf received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:37.138151 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d0bbde2432f547cbab3588c16fb623cf on tablet d0bbde2432f547cbab3588c16fb623cf with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:37.138293 2555 tablet_copy_client.cc:531] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:37.149770 2555 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:37.165915 2559 tablet_copy_client.cc:316] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:37.167608 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet e6bcce456c08487491b881518f4f8b46 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:37.168108 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet e6bcce456c08487491b881518f4f8b46 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-e6bcce456c08487491b881518f4f8b46
I20260927 23:30:37.176713 1598 tablet_copy_source_session.cc:205] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:37.179559 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bcce456c08487491b881518f4f8b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:37.203701 2559 tablet_copy_client.cc:791] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:37.204290 2559 tablet_copy_client.cc:655] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:37.208953 2559 tablet_copy_client.cc:531] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:37.209087 2556 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:37.209990 2556 tablet_bootstrap.cc:492] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:37.210716 2556 ts_tablet_manager.cc:1397] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.090s user 0.025s sys 0.025s
I20260927 23:30:37.213719 2556 raft_consensus.cc:348] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.216329 2556 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:37.218756 2556 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.222296 2556 ts_tablet_manager.cc:1428] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 23:30:37.223704 2555 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:37.224550 2555 tablet_bootstrap.cc:492] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:37.224674 2559 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:37.225203 2555 ts_tablet_manager.cc:1397] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.006s
I20260927 23:30:37.228615 2555 raft_consensus.cc:348] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.229429 2555 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:37.230211 2555 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.231272 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f575009df7d846d997a605fb1e38ad31 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:37.231729 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f575009df7d846d997a605fb1e38ad31 on tablet f575009df7d846d997a605fb1e38ad31 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:37.232822 2555 ts_tablet_manager.cc:1428] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260927 23:30:37.234516 1747 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e412df23ce8e4e71bc3c5bc171ca0e85 received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:37.234884 1747 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e412df23ce8e4e71bc3c5bc171ca0e85 on tablet e412df23ce8e4e71bc3c5bc171ca0e85 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:37.253600 2559 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:37.255225 2559 tablet_bootstrap.cc:492] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:37.255888 2559 ts_tablet_manager.cc:1397] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.011s
I20260927 23:30:37.258293 2559 raft_consensus.cc:348] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.259027 2559 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:37.259611 2559 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:37.261929 2559 ts_tablet_manager.cc:1428] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260927 23:30:37.263411 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e6bcce456c08487491b881518f4f8b46 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:37.263787 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-e6bcce456c08487491b881518f4f8b46 on tablet e6bcce456c08487491b881518f4f8b46 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:37.461561 2293 raft_consensus.cc:1203] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:37.538451 2293 raft_consensus.cc:1203] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:37.553079 2293 raft_consensus.cc:1203] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:37.665030 2293 raft_consensus.cc:1203] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:37.851349 2545 raft_consensus.cc:1050] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:37.853480 2545 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:37.859310 2293 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:37.859831 1737 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:37.860810 1659 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:37.860932 2568 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:37.862272 2546 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:37.865720 2545 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:37.896476 2545 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.897892 1737 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.904616 2293 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.914786 1659 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.917225 1485 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.960709 1580 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:37.967285 2293 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:37.970248 2545 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:37.973776 1734 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:37.974923 2545 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:37.976629 2546 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.978534 2293 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.993208 1483 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.994599 1734 raft_consensus.cc:2932] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:37.995950 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e6bcce456c08487491b881518f4f8b46 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:38.014950 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet e6bcce456c08487491b881518f4f8b46 with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:38.029868 2576 tablet_replica.cc:330] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:38.031716 2576 raft_consensus.cc:2229] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:38.032501 2576 raft_consensus.cc:2258] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:38.035429 2576 ts_tablet_manager.cc:1905] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:38.055449 2576 ts_tablet_manager.cc:1918] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:38.055832 2576 log.cc:1196] T e6bcce456c08487491b881518f4f8b46 P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/e6bcce456c08487491b881518f4f8b46
I20260927 23:30:38.057354 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet e6bcce456c08487491b881518f4f8b46 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:38.111044 2545 raft_consensus.cc:1050] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:38.113221 2545 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.119010 2293 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.119393 1659 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.120050 1734 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.120527 2545 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.121464 2546 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.122306 2397 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.136307 2397 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.138414 1659 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.148211 2291 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.150892 2530 raft_consensus.cc:1050] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:38.153177 2530 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.148171 1734 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.158452 1483 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:38.173518 2513 raft_consensus.cc:1050] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:38.175565 2513 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.188194 1586 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.191962 2291 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.193921 2564 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:38.204536 1657 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.205992 2293 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.206429 1580 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.207453 2397 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:38.208118 2564 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.209653 2529 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.210342 1657 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.212114 2546 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:38.218883 2397 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.221594 2293 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.223693 2529 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.230829 1659 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.232352 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f575009df7d846d997a605fb1e38ad31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:38.235466 1482 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.249264 1734 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.251838 1580 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.251315 2581 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:38.253875 1656 raft_consensus.cc:2932] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.258448 2291 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:38.262519 2435 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:38.265389 2513 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:38.282028 1483 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:38.292469 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet f575009df7d846d997a605fb1e38ad31 with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:38.251873 2293 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.313668 2592 tablet_replica.cc:330] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
W20260927 23:30:38.319597 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56983.3us
I20260927 23:30:38.318106 2292 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.315914 2588 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.347213 1580 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.355260 2592 raft_consensus.cc:2229] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:38.354135 1733 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.360846 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.372347 2592 raft_consensus.cc:2258] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:38.379223 2592 ts_tablet_manager.cc:1905] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:38.400204 1734 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.402412 1579 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.417528 2293 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.418720 1659 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.419263 2592 ts_tablet_manager.cc:1918] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:38.419752 2592 log.cc:1196] T f575009df7d846d997a605fb1e38ad31 P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/f575009df7d846d997a605fb1e38ad31
I20260927 23:30:38.419499 2598 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:38.421157 2529 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:38.421458 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet f575009df7d846d997a605fb1e38ad31 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:38.425850 2598 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.428463 2292 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.432235 1656 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:38.440076 2291 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.440076 1586 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:38.442214 2588 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:38.447393 2588 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:38.447844 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e412df23ce8e4e71bc3c5bc171ca0e85 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:38.450595 2513 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.457440 1656 raft_consensus.cc:2932] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.459403 1482 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.464190 1586 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.464399 2291 raft_consensus.cc:2932] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.478688 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d0bbde2432f547cbab3588c16fb623cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:38.478735 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet e412df23ce8e4e71bc3c5bc171ca0e85 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:38.482548 2599 tablet_replica.cc:330] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:38.483716 2599 raft_consensus.cc:2229] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:38.487557 2599 raft_consensus.cc:2258] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:38.489379 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:38.495975 2599 ts_tablet_manager.cc:1905] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:38.502113 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet d0bbde2432f547cbab3588c16fb623cf with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:38.503142 2592 tablet_replica.cc:330] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:38.504746 2592 raft_consensus.cc:2229] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:38.505465 2592 raft_consensus.cc:2258] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:38.508565 2592 ts_tablet_manager.cc:1905] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:38.515758 2599 ts_tablet_manager.cc:1918] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:38.516295 2599 log.cc:1196] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/e412df23ce8e4e71bc3c5bc171ca0e85
I20260927 23:30:38.518692 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet e412df23ce8e4e71bc3c5bc171ca0e85 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:38.527009 2592 ts_tablet_manager.cc:1918] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:38.527376 2592 log.cc:1196] T d0bbde2432f547cbab3588c16fb623cf P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/d0bbde2432f547cbab3588c16fb623cf
I20260927 23:30:38.528630 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet d0bbde2432f547cbab3588c16fb623cf (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:39.555815 1733 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
W20260927 23:30:39.564308 1703 consensus_peers.cc:487] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b90254fab234e29ba132c165c446490. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:39.564917 1586 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:39.565686 1657 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:39.566414 2600 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.567615 2564 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.574847 2600 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.577184 1656 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.584312 1586 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.590202 1729 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:39.593091 1482 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:39.610563 1586 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:39.610790 1657 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:39.611011 1703 consensus_peers.cc:487] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87786104826046e7852edc590e194f5d. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:39.612128 2600 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.613845 2598 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.620993 2600 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.622774 1586 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.634779 1657 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.643664 1733 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:39.649783 1484 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.665030 1657 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:39.665205 1703 consensus_peers.cc:487] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9fe23b5411b441fbd8792ef9dc4e99b. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:39.667299 1579 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:39.671681 2564 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.675987 2598 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.681740 2564 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.684563 1586 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.687062 1657 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.694957 1580 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:39.701327 1483 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:39.713214 1657 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:39.715052 1554 consensus_peers.cc:487] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d069fe150057489e8a595e9fe7c32de5. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:39.715366 1733 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:39.716558 2568 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.718344 2546 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:39.727366 2568 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.729135 1656 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.735451 1733 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:39.751346 1482 catalog_manager.cc:5495] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:40.098985 2620 ts_tablet_manager.cc:927] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:40.100764 2620 tablet_copy_client.cc:316] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:40.102223 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet d069fe150057489e8a595e9fe7c32de5 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:40.102752 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet d069fe150057489e8a595e9fe7c32de5 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-d069fe150057489e8a595e9fe7c32de5
I20260927 23:30:40.109054 1598 tablet_copy_source_session.cc:205] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:40.111856 2620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d069fe150057489e8a595e9fe7c32de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:40.124727 2620 tablet_copy_client.cc:791] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:40.125376 2620 tablet_copy_client.cc:655] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:40.133935 2620 tablet_copy_client.cc:531] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:40.143987 2623 ts_tablet_manager.cc:927] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:40.148221 2623 tablet_copy_client.cc:316] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:40.149746 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet b9fe23b5411b441fbd8792ef9dc4e99b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:40.150483 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet b9fe23b5411b441fbd8792ef9dc4e99b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-b9fe23b5411b441fbd8792ef9dc4e99b
I20260927 23:30:40.153083 2625 ts_tablet_manager.cc:927] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:40.155656 2622 ts_tablet_manager.cc:927] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:40.158281 1747 tablet_copy_source_session.cc:205] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:40.159829 2620 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:40.161727 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe23b5411b441fbd8792ef9dc4e99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:40.176744 2625 tablet_copy_client.cc:316] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:40.178277 1747 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet 2b90254fab234e29ba132c165c446490 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:40.178819 1747 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet 2b90254fab234e29ba132c165c446490 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-2b90254fab234e29ba132c165c446490
I20260927 23:30:40.185415 2622 tablet_copy_client.cc:316] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:35731
I20260927 23:30:40.187665 1746 tablet_copy_service.cc:140] P a2a3317e8da742d9aefe4c251abd4566: Received BeginTabletCopySession request for tablet 87786104826046e7852edc590e194f5d from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:60802)
I20260927 23:30:40.188480 1746 tablet_copy_service.cc:161] P a2a3317e8da742d9aefe4c251abd4566: Beginning new tablet copy session on tablet 87786104826046e7852edc590e194f5d from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:60802: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-87786104826046e7852edc590e194f5d
I20260927 23:30:40.188691 1747 tablet_copy_source_session.cc:205] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:40.191557 2623 tablet_copy_client.cc:791] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:40.192132 2623 tablet_copy_client.cc:655] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:40.197840 2625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b90254fab234e29ba132c165c446490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:40.198900 2623 tablet_copy_client.cc:531] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:40.205771 2620 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:40.206696 2620 tablet_bootstrap.cc:492] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:40.207455 2620 ts_tablet_manager.cc:1397] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.016s
I20260927 23:30:40.220554 1746 tablet_copy_source_session.cc:205] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:40.220950 2623 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:40.222110 2625 tablet_copy_client.cc:791] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:40.210157 2620 raft_consensus.cc:348] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.222658 2620 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:40.223297 2620 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.224879 2622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87786104826046e7852edc590e194f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:40.224900 2625 tablet_copy_client.cc:655] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:40.235184 2625 tablet_copy_client.cc:531] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:40.235865 2620 ts_tablet_manager.cc:1428] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.028s user 0.010s sys 0.003s
I20260927 23:30:40.237380 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d069fe150057489e8a595e9fe7c32de5 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:40.237764 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d069fe150057489e8a595e9fe7c32de5 on tablet d069fe150057489e8a595e9fe7c32de5 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:40.254464 2625 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:40.256253 2622 tablet_copy_client.cc:791] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:40.256819 2622 tablet_copy_client.cc:655] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:40.260058 2623 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:40.260867 2623 tablet_bootstrap.cc:492] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:40.261166 2622 tablet_copy_client.cc:531] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:40.261461 2623 ts_tablet_manager.cc:1397] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.000s
I20260927 23:30:40.264127 2623 raft_consensus.cc:348] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.264916 2623 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:40.265522 2623 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.267906 2623 ts_tablet_manager.cc:1428] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 23:30:40.273746 2622 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:40.275646 1746 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-b9fe23b5411b441fbd8792ef9dc4e99b received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:40.276099 1746 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-b9fe23b5411b441fbd8792ef9dc4e99b on tablet b9fe23b5411b441fbd8792ef9dc4e99b with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:40.302135 2625 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:40.303144 2625 tablet_bootstrap.cc:492] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:40.303839 2625 ts_tablet_manager.cc:1397] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.050s user 0.028s sys 0.002s
I20260927 23:30:40.305651 2622 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:40.306535 2625 raft_consensus.cc:348] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.307410 2622 tablet_bootstrap.cc:492] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:40.307535 2625 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:40.309437 2622 ts_tablet_manager.cc:1397] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.015s
I20260927 23:30:40.309028 2625 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.311156 2625 ts_tablet_manager.cc:1428] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:30:40.313838 2622 raft_consensus.cc:348] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.314849 2622 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:40.315493 2622 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:40.321758 1746 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2b90254fab234e29ba132c165c446490 received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:40.322187 1746 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-2b90254fab234e29ba132c165c446490 on tablet 2b90254fab234e29ba132c165c446490 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:40.322551 2622 ts_tablet_manager.cc:1428] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 23:30:40.324636 1746 tablet_copy_service.cc:342] P a2a3317e8da742d9aefe4c251abd4566: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-87786104826046e7852edc590e194f5d received from {username='slave'} at 127.0.0.1:60802
I20260927 23:30:40.325002 1746 tablet_copy_service.cc:433] P a2a3317e8da742d9aefe4c251abd4566: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-87786104826046e7852edc590e194f5d on tablet 87786104826046e7852edc590e194f5d with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:40.521162 2293 raft_consensus.cc:1203] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:40.585421 2293 raft_consensus.cc:1203] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:40.601399 2293 raft_consensus.cc:1203] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:40.769075 2293 raft_consensus.cc:1203] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:41.082981 2633 raft_consensus.cc:1050] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:41.085129 2633 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.091032 2291 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.091257 1657 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.092610 2564 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.093170 1586 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.094012 2612 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.095794 2633 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.112685 2633 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.114904 1586 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.115257 1657 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.121762 2293 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.127487 2619 raft_consensus.cc:1050] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:41.129531 2619 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.136502 1484 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.154865 2564 raft_consensus.cc:1050] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:41.157121 2564 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.163697 1657 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.165278 2633 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.159577 2293 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.169080 2568 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:41.171566 2293 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.175263 1656 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.175972 2564 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.176821 1733 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.176678 2619 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.181249 1580 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.181982 2619 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.187394 2612 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:41.218662 2640 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.224992 1586 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.237238 1657 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.246387 2568 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.267621 1733 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.272919 2293 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.276324 2633 raft_consensus.cc:1050] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:41.278690 2633 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.297293 2291 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.326367 1657 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.333470 1656 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.335027 2633 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.347831 1733 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:41.360569 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 58490.6us
I20260927 23:30:41.361029 1482 catalog_manager.cc:5495] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.363899 1657 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.364609 2291 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.366251 2564 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.362689 1482 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.374346 2291 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.377349 2633 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.377094 1656 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.379541 2633 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.375938 2657 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.384481 1579 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:41.387727 2640 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:41.394133 2293 raft_consensus.cc:2932] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.416175 1733 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.424098 1659 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.426386 2640 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:41.427441 2293 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.431231 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b9fe23b5411b441fbd8792ef9dc4e99b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:41.429163 2640 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.436486 1579 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.438359 2564 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.440232 2656 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.444787 1483 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.446864 1659 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.449812 2293 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.424316 1586 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.471948 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2b90254fab234e29ba132c165c446490 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:41.469692 2291 raft_consensus.cc:2932] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.479873 2292 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.481318 2546 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.488646 1733 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.489951 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet b9fe23b5411b441fbd8792ef9dc4e99b with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:41.491017 1485 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.494976 2546 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.496927 2546 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.497779 1484 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.500181 2291 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.503160 2662 tablet_replica.cc:330] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:41.504619 2662 raft_consensus.cc:2229] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:41.502553 1729 raft_consensus.cc:2932] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.509289 2662 raft_consensus.cc:2258] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:41.485356 1659 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.512096 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d069fe150057489e8a595e9fe7c32de5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:41.515838 2662 ts_tablet_manager.cc:1905] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:41.522564 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b90254fab234e29ba132c165c446490 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:41.530560 1484 catalog_manager.cc:5495] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:41.547796 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet d069fe150057489e8a595e9fe7c32de5 with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:41.555891 2662 ts_tablet_manager.cc:1918] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:41.556447 2662 log.cc:1196] T b9fe23b5411b441fbd8792ef9dc4e99b P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/b9fe23b5411b441fbd8792ef9dc4e99b
I20260927 23:30:41.558624 2662 tablet_replica.cc:330] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:41.559023 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet b9fe23b5411b441fbd8792ef9dc4e99b (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:41.559681 2662 raft_consensus.cc:2229] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:41.560437 2662 raft_consensus.cc:2258] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:41.567226 2662 ts_tablet_manager.cc:1905] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:41.573024 1734 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:41.579389 2291 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.580646 2657 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.581094 1586 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:41.582780 2564 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:41.589953 2656 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.592881 1586 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.606966 2291 raft_consensus.cc:2932] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.610235 2664 tablet_replica.cc:330] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:41.611219 2664 raft_consensus.cc:2229] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:41.611742 2662 ts_tablet_manager.cc:1918] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:41.611863 2664 raft_consensus.cc:2258] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:41.612257 2662 log.cc:1196] T 2b90254fab234e29ba132c165c446490 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/2b90254fab234e29ba132c165c446490
I20260927 23:30:41.615744 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 2b90254fab234e29ba132c165c446490 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:41.617161 1483 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:41.618763 1471 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 87786104826046e7852edc590e194f5d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:41.621227 2664 ts_tablet_manager.cc:1905] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:41.630689 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 87786104826046e7852edc590e194f5d with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:41.642639 2664 ts_tablet_manager.cc:1918] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:41.643115 2664 log.cc:1196] T d069fe150057489e8a595e9fe7c32de5 P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/d069fe150057489e8a595e9fe7c32de5
I20260927 23:30:41.644775 2664 tablet_replica.cc:330] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:41.644716 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet d069fe150057489e8a595e9fe7c32de5 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:41.645753 2664 raft_consensus.cc:2229] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:41.646188 2664 raft_consensus.cc:2258] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:41.648241 2664 ts_tablet_manager.cc:1905] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:41.658226 2664 ts_tablet_manager.cc:1918] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:41.658632 2664 log.cc:1196] T 87786104826046e7852edc590e194f5d P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/87786104826046e7852edc590e194f5d
I20260927 23:30:41.659765 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 87786104826046e7852edc590e194f5d (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:42.700706 1581 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
W20260927 23:30:42.708212 1554 consensus_peers.cc:487] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cfb0bd166774a248820c1b90412697d. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:42.708597 1656 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.709002 1734 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.709751 2654 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:42.710590 2546 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:42.719728 2654 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.732079 1659 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.735024 1733 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.742290 1485 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:42.748117 1581 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
W20260927 23:30:42.766907 1554 consensus_peers.cc:487] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58c5d7cee3a4fa7b7b7d55bf8cd213b. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:42.767659 1733 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.780393 2670 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:42.785153 1659 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.786413 2670 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:42.792950 2546 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.800349 1733 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.800925 1659 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.805419 1656 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:42.819048 1483 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
W20260927 23:30:42.820271 1629 consensus_peers.cc:487] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 861e779ad2d04547bd9be0bfbc2616c5. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:42.820518 1733 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.821684 2588 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:42.836653 1581 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.838526 2681 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:42.845652 2588 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.847296 1734 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.848471 1581 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.857118 1656 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:42.862032 1484 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:42.887952 1579 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:42.888164 1629 consensus_peers.cc:487] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07438388789747e68da9846e153a903b. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:42.888365 1733 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:42.890336 2665 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:42.892208 2681 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:42.936290 2665 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.937813 1733 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.953883 1483 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:42.957490 1579 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:43.246857 2688 ts_tablet_manager.cc:927] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:43.255249 2688 tablet_copy_client.cc:316] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:43.256940 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 861e779ad2d04547bd9be0bfbc2616c5 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:43.257467 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 861e779ad2d04547bd9be0bfbc2616c5 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-861e779ad2d04547bd9be0bfbc2616c5
I20260927 23:30:43.265009 1673 tablet_copy_source_session.cc:205] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:43.268837 2688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861e779ad2d04547bd9be0bfbc2616c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:43.270277 2690 ts_tablet_manager.cc:927] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:43.272554 2690 tablet_copy_client.cc:316] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:43.274050 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:43.274763 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-d58c5d7cee3a4fa7b7b7d55bf8cd213b
I20260927 23:30:43.284165 1598 tablet_copy_source_session.cc:205] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:43.286751 2690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:43.291328 2688 tablet_copy_client.cc:791] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:43.292487 2688 tablet_copy_client.cc:655] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:43.298780 2688 tablet_copy_client.cc:531] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:43.302459 2690 tablet_copy_client.cc:791] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:43.303148 2690 tablet_copy_client.cc:655] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:43.306933 2690 tablet_copy_client.cc:531] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:43.310041 2688 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:43.326166 2690 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:43.334342 2693 ts_tablet_manager.cc:927] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:43.336956 2688 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:43.338016 2688 tablet_bootstrap.cc:492] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:43.338814 2688 ts_tablet_manager.cc:1397] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20260927 23:30:43.341856 2688 raft_consensus.cc:348] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.343272 2693 tablet_copy_client.cc:316] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:43.344187 2688 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:43.345041 2688 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.350725 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 4cfb0bd166774a248820c1b90412697d from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:43.351302 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 4cfb0bd166774a248820c1b90412697d from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-4cfb0bd166774a248820c1b90412697d
I20260927 23:30:43.360317 1598 tablet_copy_source_session.cc:205] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:43.361075 2690 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:43.363023 2690 tablet_bootstrap.cc:492] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:43.363667 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfb0bd166774a248820c1b90412697d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:43.363806 2690 ts_tablet_manager.cc:1397] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.038s user 0.028s sys 0.004s
I20260927 23:30:43.367774 2690 raft_consensus.cc:348] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.368769 2690 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:43.369511 2690 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.379707 2688 ts_tablet_manager.cc:1428] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.040s user 0.010s sys 0.003s
I20260927 23:30:43.391640 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-861e779ad2d04547bd9be0bfbc2616c5 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:43.392093 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-861e779ad2d04547bd9be0bfbc2616c5 on tablet 861e779ad2d04547bd9be0bfbc2616c5 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:43.393105 2690 ts_tablet_manager.cc:1428] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.029s user 0.015s sys 0.011s
I20260927 23:30:43.395017 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d58c5d7cee3a4fa7b7b7d55bf8cd213b received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:43.395481 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-d58c5d7cee3a4fa7b7b7d55bf8cd213b on tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:43.404256 2693 tablet_copy_client.cc:791] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:43.404867 2693 tablet_copy_client.cc:655] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:43.408221 2693 tablet_copy_client.cc:531] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:43.416553 2693 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:43.434463 2693 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:43.435282 2693 tablet_bootstrap.cc:492] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:43.436003 2693 ts_tablet_manager.cc:1397] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.006s
I20260927 23:30:43.438546 2693 raft_consensus.cc:348] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.439185 2693 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:43.439842 2693 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.445621 2693 ts_tablet_manager.cc:1428] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 23:30:43.448585 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-4cfb0bd166774a248820c1b90412697d received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:43.448967 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-4cfb0bd166774a248820c1b90412697d on tablet 4cfb0bd166774a248820c1b90412697d with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:43.484499 2693 ts_tablet_manager.cc:927] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:43.486055 2693 tablet_copy_client.cc:316] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:43.487387 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 07438388789747e68da9846e153a903b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:43.487933 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 07438388789747e68da9846e153a903b from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-07438388789747e68da9846e153a903b
I20260927 23:30:43.495030 1673 tablet_copy_source_session.cc:205] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:43.497737 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07438388789747e68da9846e153a903b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:43.511168 2693 tablet_copy_client.cc:791] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:43.511680 2693 tablet_copy_client.cc:655] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:43.514705 2693 tablet_copy_client.cc:531] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:43.521735 2693 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:43.540652 2693 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:43.541525 2693 tablet_bootstrap.cc:492] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:43.542167 2693 ts_tablet_manager.cc:1397] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20260927 23:30:43.545061 2693 raft_consensus.cc:348] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.545975 2693 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:43.546691 2693 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:43.549228 2693 ts_tablet_manager.cc:1428] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260927 23:30:43.552188 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-07438388789747e68da9846e153a903b received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:43.552597 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-07438388789747e68da9846e153a903b on tablet 07438388789747e68da9846e153a903b with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:43.659183 2291 raft_consensus.cc:1203] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:43.701331 2291 raft_consensus.cc:1203] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:43.878559 2293 raft_consensus.cc:1203] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:43.992101 2293 raft_consensus.cc:1203] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:44.201159 2546 raft_consensus.cc:1050] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:44.203147 2546 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.208720 1733 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.208841 2291 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.210162 2667 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.211259 2670 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.213719 1659 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.217454 2546 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.244343 2703 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.247653 1656 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.249245 2291 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.259861 1734 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.268031 1484 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:44.295497 2685 raft_consensus.cc:1050] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:44.297853 2685 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:44.342221 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56778.3us
W20260927 23:30:44.361003 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 60269.1us
I20260927 23:30:44.366932 2293 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.369180 2588 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.371358 1586 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.372074 1734 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.376125 2588 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.380043 2670 raft_consensus.cc:1050] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:44.381338 1581 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.383039 2670 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.384155 2665 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:44.389312 2293 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.390861 2546 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.401206 1734 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.400441 2702 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.402684 2546 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.402782 2588 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.401919 2293 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.410220 2291 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.419761 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:44.428028 1656 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.430925 2702 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.428694 1734 raft_consensus.cc:2932] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.436064 2292 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.437729 2702 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.442912 1733 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.428978 1482 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:44.447199 2670 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.457322 2667 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:44.457865 2292 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.458946 1733 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.486050 2708 raft_consensus.cc:1050] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:44.488355 2708 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.488068 1656 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.490747 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:30:44.495729 1484 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:44.519722 1737 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.545990 1482 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.552255 1656 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.553453 2723 tablet_replica.cc:330] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:30:44.554461 2723 raft_consensus.cc:2229] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:44.555100 2723 raft_consensus.cc:2258] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:44.555526 1581 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.558985 2723 ts_tablet_manager.cc:1905] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:44.563930 1579 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:44.569801 1733 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.571305 2588 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.575888 2292 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.577363 2723 ts_tablet_manager.cc:1918] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:44.577512 2667 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.577869 2723 log.cc:1196] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/d58c5d7cee3a4fa7b7b7d55bf8cd213b
I20260927 23:30:44.582684 2293 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.590812 2546 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.598376 2293 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.600054 1581 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.600680 1733 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.602201 2711 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:44.605376 2711 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:44.620672 1729 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.631747 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet d58c5d7cee3a4fa7b7b7d55bf8cd213b (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:44.635805 1660 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:44.644423 2588 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.647905 2292 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:44.656113 2727 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:30:44.658511 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4cfb0bd166774a248820c1b90412697d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:44.658936 2708 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:44.668713 2293 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.604182 2725 raft_consensus.cc:2932] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.692828 1483 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:44.783347 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 127629us
I20260927 23:30:44.786703 1660 raft_consensus.cc:2932] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.806684 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cfb0bd166774a248820c1b90412697d with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:44.810961 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 861e779ad2d04547bd9be0bfbc2616c5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:44.813443 1482 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.818143 2711 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.857134 2731 tablet_replica.cc:330] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:44.858466 2731 raft_consensus.cc:2229] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:44.859124 2731 raft_consensus.cc:2258] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:44.896092 1579 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.926177 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 861e779ad2d04547bd9be0bfbc2616c5 with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:30:44.930143 2733 tablet_replica.cc:330] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:30:44.931245 2733 raft_consensus.cc:2229] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:44.931879 2733 raft_consensus.cc:2258] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:44.944676 2292 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:44.962077 2731 ts_tablet_manager.cc:1905] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:44.972908 2733 ts_tablet_manager.cc:1905] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:44.987218 2733 ts_tablet_manager.cc:1918] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:44.988828 2733 log.cc:1196] T 861e779ad2d04547bd9be0bfbc2616c5 P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/861e779ad2d04547bd9be0bfbc2616c5
I20260927 23:30:44.992558 2731 ts_tablet_manager.cc:1918] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:44.992986 2731 log.cc:1196] T 4cfb0bd166774a248820c1b90412697d P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/4cfb0bd166774a248820c1b90412697d
I20260927 23:30:45.024214 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 861e779ad2d04547bd9be0bfbc2616c5 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:30:45.071259 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet 4cfb0bd166774a248820c1b90412697d (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
W20260927 23:30:45.081768 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 88610.9us
I20260927 23:30:45.087116 1733 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:45.099187 1483 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:45.128705 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 47617.8us
I20260927 23:30:45.145196 1663 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:45.167520 2293 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:45.170773 2711 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:45.182071 1582 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:45.188388 2732 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:45.210057 2725 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:30:45.205900 2293 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:45.226847 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 07438388789747e68da9846e153a903b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:30:45.229480 1582 raft_consensus.cc:2932] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:45.252980 1485 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index 3 to 4, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:45.274487 1720 tablet_service.cc:1514] Processing DeleteTablet for tablet 07438388789747e68da9846e153a903b with delete_type TABLET_DATA_TOMBSTONED (TS a2a3317e8da742d9aefe4c251abd4566 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38800
I20260927 23:30:45.275794 2731 tablet_replica.cc:330] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: stopping tablet replica
I20260927 23:30:45.276945 2731 raft_consensus.cc:2229] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:30:45.277851 2731 raft_consensus.cc:2258] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:30:45.281071 2731 ts_tablet_manager.cc:1905] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:30:45.293705 2731 ts_tablet_manager.cc:1918] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:30:45.294062 2731 log.cc:1196] T 07438388789747e68da9846e153a903b P a2a3317e8da742d9aefe4c251abd4566: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-2-root/wals/07438388789747e68da9846e153a903b
I20260927 23:30:45.295498 1471 catalog_manager.cc:4841] TS a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): tablet 07438388789747e68da9846e153a903b (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
W20260927 23:30:45.849050 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002d52e0 after lost signal to thread 399
W20260927 23:30:45.850421 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002464c0 after lost signal to thread 1520
W20260927 23:30:45.851845 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08001688e0 after lost signal to thread 1612
W20260927 23:30:45.853097 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08000f2340 after lost signal to thread 1687
W20260927 23:30:45.854249 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002b9bc0 after lost signal to thread 1761
W20260927 23:30:45.855705 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002c48a0 after lost signal to thread 2318
I20260927 23:30:46.029004 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:46.105492 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:46.106185 1580 raft_consensus.cc:595] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:46.125656 1656 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:46.126294 1656 raft_consensus.cc:595] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:46.128966 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:46.129550 1580 raft_consensus.cc:595] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:46.132072 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:46.132598 1580 raft_consensus.cc:595] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:46.138044 2727 raft_consensus.cc:979] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.138525 2727 raft_consensus.cc:1067] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.140563 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48296
I20260927 23:30:46.141122 2292 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.141496 2292 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.150667 2292 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.154028 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.154768 1580 raft_consensus.cc:3032] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.154821 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.155575 1729 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.155136 1580 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.159114 1580 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.160079 1580 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.164497 1729 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.167490 2292 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.168644 2258 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:46.170300 1729 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87786104826046e7852edc590e194f5d"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:46.170936 1729 raft_consensus.cc:595] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:46.171955 1580 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.175195 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:46.175621 2745 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.175693 1580 raft_consensus.cc:595] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:46.178442 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:46.178925 1580 raft_consensus.cc:595] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:46.180368 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260927 23:30:46.180482 2745 raft_consensus.cc:686] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:46.181793 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:46.181528 2745 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.182474 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:46.202211 1485 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:46.221516 2738 raft_consensus.cc:979] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:46.222011 2738 raft_consensus.cc:1067] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:46.239836 1656 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:30:46.240509 1656 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.240808 1656 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.250932 1656 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.257854 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.258417 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.259166 1729 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.258440 1580 raft_consensus.cc:3032] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.269302 1580 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.270054 1580 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.271173 1580 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.295150 1729 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:46.298552 1627 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:46.299481 2732 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.299934 2732 raft_consensus.cc:686] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:46.300292 2750 raft_consensus.cc:979] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:46.300782 2750 raft_consensus.cc:1067] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:46.300766 2732 consensus_queue.cc:237] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.302809 1660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:30:46.303277 1660 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.303580 1660 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.310218 1580 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:46.310505 1660 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.310922 1656 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.335709 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.446763 1485 catalog_manager.cc:5495] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:46.463848 2757 raft_consensus.cc:979] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: : Instructing follower 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:46.465267 2757 raft_consensus.cc:1067] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Signalling peer 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:46.464563 1660 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.465969 1582 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.470597 1582 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.472746 1582 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.485628 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.486265 1733 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.491417 1579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:46518
I20260927 23:30:46.491974 1579 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.492326 1579 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.503693 1733 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:46.515347 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:46.506740 1579 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.516449 2758 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.516948 2758 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:46.517876 2758 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.519380 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.520015 1733 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.524431 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:46.525070 1660 raft_consensus.cc:3032] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.525413 1660 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:46.526151 1660 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:46.527268 1660 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.535302 1733 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:46.535957 1579 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.540367 1660 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:46.552789 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:46.553679 2760 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.554147 2760 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.554952 2760 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:46.664512 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51720.3us
I20260927 23:30:46.665648 1582 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.678725 1729 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:46.682276 1579 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:46.683637 2745 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:46.686630 2755 raft_consensus.cc:979] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.687184 2710 raft_consensus.cc:1067] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.688577 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87786104826046e7852edc590e194f5d"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:51538
I20260927 23:30:46.689074 2292 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.689409 2292 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.695815 2292 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.696571 1582 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:46.698535 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.699043 1582 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.701278 2292 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.701844 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.702785 1729 raft_consensus.cc:3032] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.703122 1729 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:46.703887 1729 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.705327 1729 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.723699 1485 catalog_manager.cc:5495] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:46.728752 1582 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.729036 1729 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.729893 2259 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:46.742759 2745 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.743323 2745 raft_consensus.cc:686] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:46.747191 2745 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.759732 1483 catalog_manager.cc:5495] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:46.766718 1729 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:46.769536 2745 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:46.774694 1663 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:46.775454 2760 raft_consensus.cc:979] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.777444 2760 raft_consensus.cc:1067] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
W20260927 23:30:46.788808 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59133.5us
W20260927 23:30:46.809203 1629 consensus_peers.cc:487] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ddb51781fec46b194174de5f4406def. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:46.812568 1737 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:46.814155 2758 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:46.821211 1729 raft_consensus.cc:1261] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:46.821705 1579 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:46.822896 2758 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:46.826906 2756 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:46.827936 1729 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:46.833317 1484 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:46.834457 1579 raft_consensus.cc:1261] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:46.836519 2754 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:46.836972 2758 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:46.840843 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48296
I20260927 23:30:46.841396 2292 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.841768 2292 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.857178 2292 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.870177 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.870805 1582 raft_consensus.cc:3032] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.871162 1582 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.871337 2770 raft_consensus.cc:979] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.872018 2772 raft_consensus.cc:1067] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:46.871989 1582 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.878660 2292 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.879638 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.913439 1729 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.915422 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48296
I20260927 23:30:46.915974 2292 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:46.916301 2292 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.917053 1580 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:46.873087 1579 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:46.918249 1582 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.928794 1482 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:46.931645 1729 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.932668 2258 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:46.935478 2780 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:46.935928 2780 raft_consensus.cc:686] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:46.936332 1582 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:46.936765 2780 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.942035 2292 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.950048 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:46.951447 1737 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.954049 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:46.955855 1733 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:46.958571 1579 raft_consensus.cc:3032] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:46.959053 1579 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:46.960191 2292 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:46.961779 2763 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260927 23:30:46.968916 1579 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:46.970043 1579 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:46.982467 1586 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:46.987712 2711 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:47.007507 1579 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.009539 1737 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.008733 2259 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:47.018513 1660 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:47.019093 1554 consensus_peers.cc:487] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0be2a211b7754af89c38527623e5d967. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:47.020080 2727 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.021131 1660 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:47.022754 1737 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:47.023573 2770 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.023995 2727 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.024147 1486 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:47.022281 2780 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.079927 1733 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:47.091432 2754 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260927 23:30:47.074301 2780 raft_consensus.cc:686] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:30:47.107298 2792 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260927 23:30:47.107813 2780 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.170292 2769 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.170717 1663 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.175936 1586 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.196570 1483 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:30:47.209472 2794 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260927 23:30:47.209478 2776 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260927 23:30:47.219808 1484 catalog_manager.cc:5495] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:47.208657 1586 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.246814 1734 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.265935 1729 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:47.267181 2763 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.268805 1656 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 23:30:47.277861 1554 consensus_peers.cc:487] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f662be9387d148199907fd0e049ba4be. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:47.282404 2738 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.295758 2738 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.298591 1729 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.319182 1484 catalog_manager.cc:5495] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.353241 1729 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.355943 2775 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:47.359675 1734 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.362871 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:47.363183 2776 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:47.381705 1579 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.383836 1662 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.384956 2780 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:47.421450 2809 ts_tablet_manager.cc:927] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:47.422133 2808 ts_tablet_manager.cc:927] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:47.424888 1660 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.430644 2808 tablet_copy_client.cc:316] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:47.435572 2809 tablet_copy_client.cc:316] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:47.437572 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet 0be2a211b7754af89c38527623e5d967 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:47.438095 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet 0be2a211b7754af89c38527623e5d967 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-0be2a211b7754af89c38527623e5d967
I20260927 23:30:47.450760 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 5ddb51781fec46b194174de5f4406def from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:47.451366 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 5ddb51781fec46b194174de5f4406def from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-5ddb51781fec46b194174de5f4406def
I20260927 23:30:47.461083 1598 tablet_copy_source_session.cc:205] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:47.475939 1673 tablet_copy_source_session.cc:205] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:47.476640 1729 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:47.479189 2732 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.493490 2808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddb51781fec46b194174de5f4406def. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:30:47.494266 1629 consensus_peers.cc:487] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 -> Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Couldn't send request to peer 0ebfcbb9c0574725a9df6ed2bf33ff47. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c719671f35640549c1b944112b3fc03. This is attempt 1: this message will repeat every 5th retry.
I20260927 23:30:47.497545 1579 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:30:47.506989 2732 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.510824 2808 tablet_copy_client.cc:791] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:47.511488 2808 tablet_copy_client.cc:655] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:47.511823 1729 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.516038 2808 tablet_copy_client.cc:531] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:47.526626 2808 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:47.537895 2809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0be2a211b7754af89c38527623e5d967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:47.543784 2732 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:47.562155 1579 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.571962 2780 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:47.565687 1483 catalog_manager.cc:5495] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.597134 1660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "07438388789747e68da9846e153a903b"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:47.598762 1660 raft_consensus.cc:595] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:47.612510 2808 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:47.613435 2808 tablet_bootstrap.cc:492] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:47.616447 1730 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:47.616662 1580 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:47.617261 1580 raft_consensus.cc:595] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:47.617903 2776 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:47.618155 1586 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } } }
I20260927 23:30:47.614133 2808 ts_tablet_manager.cc:1397] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.088s user 0.020s sys 0.007s
I20260927 23:30:47.627975 1660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:47.628587 1660 raft_consensus.cc:595] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:47.632830 1734 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:47.632599 2808 raft_consensus.cc:348] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.633560 1734 raft_consensus.cc:595] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:47.633658 2808 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:47.634472 2808 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.638123 2808 ts_tablet_manager.cc:1428] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260927 23:30:47.639133 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:47.639725 1586 raft_consensus.cc:595] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:47.639778 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-5ddb51781fec46b194174de5f4406def received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:47.640244 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-5ddb51781fec46b194174de5f4406def on tablet 5ddb51781fec46b194174de5f4406def with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:47.644826 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f662be9387d148199907fd0e049ba4be"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:47.645470 1586 raft_consensus.cc:595] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:47.648734 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 23:30:47.650381 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:47.650969 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:47.658013 2727 raft_consensus.cc:979] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.658485 2727 raft_consensus.cc:1067] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.660609 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48296
I20260927 23:30:47.661118 2292 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:47.661476 2292 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.668336 2292 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.672032 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:47.672866 1586 raft_consensus.cc:3032] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:47.674207 1586 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:47.675006 1586 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.676107 1586 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.678050 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:47.681814 1660 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.688550 2292 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:47.694509 2772 raft_consensus.cc:979] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.695019 2763 raft_consensus.cc:1067] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.696815 2293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48296
I20260927 23:30:47.697386 2293 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:47.701618 2293 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.716521 2293 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.717527 1586 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.719756 2259 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:47.720625 2780 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.721069 2780 raft_consensus.cc:686] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:47.721889 2780 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.730767 2756 raft_consensus.cc:979] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: : Instructing follower 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:47.731369 2756 raft_consensus.cc:1067] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Signalling peer 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:47.733090 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:47.734722 1586 raft_consensus.cc:3032] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:47.737099 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:47.738698 1729 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.743836 1660 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.747769 1582 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:46518
I20260927 23:30:47.748301 1582 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:47.748641 1582 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.751948 1729 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.755103 2258 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:47.755903 2776 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.756429 2776 raft_consensus.cc:686] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:47.757292 2776 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.755890 1582 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:47.764292 2293 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:47.762099 1586 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:47.767042 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:47.766832 1586 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.767593 1662 raft_consensus.cc:3032] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:47.767865 1662 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:47.768118 1586 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.768507 1662 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:47.769510 1662 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.780087 1586 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.781221 1482 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:47.788986 1662 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:47.791493 1582 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:47.797711 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:47.798616 2763 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.799072 2763 raft_consensus.cc:686] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:47.799903 2763 consensus_queue.cc:237] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:47.797178 2809 tablet_copy_client.cc:791] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:47.801093 2809 tablet_copy_client.cc:655] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:47.804546 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:47.805137 1734 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.812855 1483 catalog_manager.cc:5495] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:47.823444 2809 tablet_copy_client.cc:531] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:47.827414 1734 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:47.862700 2292 raft_consensus.cc:1203] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:47.871412 2809 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:47.886984 2808 ts_tablet_manager.cc:927] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:47.895637 1484 catalog_manager.cc:5495] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:47.918409 2809 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:47.919195 2808 tablet_copy_client.cc:316] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:45501
I20260927 23:30:47.922827 2809 tablet_bootstrap.cc:492] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:47.926039 1598 tablet_copy_service.cc:140] P 0fe98f486bab4d0b9c421c145d83f1ce: Received BeginTabletCopySession request for tablet f662be9387d148199907fd0e049ba4be from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:56820)
I20260927 23:30:47.926403 2809 ts_tablet_manager.cc:1397] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.005s
I20260927 23:30:47.926625 1598 tablet_copy_service.cc:161] P 0fe98f486bab4d0b9c421c145d83f1ce: Beginning new tablet copy session on tablet f662be9387d148199907fd0e049ba4be from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:56820: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-f662be9387d148199907fd0e049ba4be
I20260927 23:30:47.929410 2809 raft_consensus.cc:348] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.930796 2756 raft_consensus.cc:979] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.931310 2756 raft_consensus.cc:1067] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:47.933485 2293 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "07438388789747e68da9846e153a903b"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:48306
I20260927 23:30:47.934376 2293 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:47.934760 2293 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.930824 2809 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:47.941828 2809 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.944310 2763 raft_consensus.cc:979] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:47.945508 2763 raft_consensus.cc:1067] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:47.949509 1656 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f662be9387d148199907fd0e049ba4be"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:30:47.950008 1656 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:47.950342 1656 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.951457 1598 tablet_copy_source_session.cc:205] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:47.958182 1656 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.962191 2809 ts_tablet_manager.cc:1428] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.035s user 0.003s sys 0.003s
I20260927 23:30:47.965327 1656 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:47.944535 2293 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.967164 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:47.967720 1580 raft_consensus.cc:3032] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:47.968009 1580 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:47.968590 2293 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:47.968693 1580 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.971546 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:47.972105 1656 raft_consensus.cc:3032] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:47.972510 1656 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:47.973163 1656 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:47.974309 1656 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.975670 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:47.976243 1586 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.978425 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:47.979031 1734 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.970088 1580 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:47.990146 1734 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:47.991128 1627 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:47.991940 2754 raft_consensus.cc:2781] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.992384 2754 raft_consensus.cc:686] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:47.993148 2754 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:47.994937 1586 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:47.996003 2259 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:47.996881 2776 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:47.997516 2776 raft_consensus.cc:686] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:48.000020 2776 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:48.001775 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-0be2a211b7754af89c38527623e5d967 received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:48.002194 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-0be2a211b7754af89c38527623e5d967 on tablet 0be2a211b7754af89c38527623e5d967 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:48.002893 2808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f662be9387d148199907fd0e049ba4be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:48.002941 1656 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:48.012205 1580 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:48.015866 1483 catalog_manager.cc:5495] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:48.025278 1484 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:48.040423 2809 ts_tablet_manager.cc:927] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Initiating tablet copy from peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:48.040982 2808 tablet_copy_client.cc:791] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:48.041556 2808 tablet_copy_client.cc:655] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:48.046216 2808 tablet_copy_client.cc:531] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:48.053506 2809 tablet_copy_client.cc:316] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:42051
I20260927 23:30:48.055303 1673 tablet_copy_service.cc:140] P 1b945905616b4989853b0a92bfbd6b98: Received BeginTabletCopySession request for tablet 9c719671f35640549c1b944112b3fc03 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 ({username='slave'} at 127.0.0.1:36962)
I20260927 23:30:48.055846 1673 tablet_copy_service.cc:161] P 1b945905616b4989853b0a92bfbd6b98: Beginning new tablet copy session on tablet 9c719671f35640549c1b944112b3fc03 from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 at {username='slave'} at 127.0.0.1:36962: session id = 0ebfcbb9c0574725a9df6ed2bf33ff47-9c719671f35640549c1b944112b3fc03
I20260927 23:30:48.065099 2808 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:48.072147 2759 raft_consensus.cc:979] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:48.072868 2740 raft_consensus.cc:1067] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:48.074667 2292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:51538
I20260927 23:30:48.075265 2292 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:48.075591 2292 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:48.080689 1673 tablet_copy_source_session.cc:205] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:30:48.081580 2292 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:48.092839 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:48.093415 1734 raft_consensus.cc:3032] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:48.093689 1734 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:48.094090 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:48.094795 1662 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:48.095082 2292 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:48.094368 1734 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:48.106881 1734 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:48.107314 1662 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:48.108453 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:48.109483 2780 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:48.109925 2780 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:48.111709 2780 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:48.116333 1734 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:48.119400 2809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c719671f35640549c1b944112b3fc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:30:48.129154 2808 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:48.130121 2808 tablet_bootstrap.cc:492] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:48.131016 2808 ts_tablet_manager.cc:1397] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.066s user 0.025s sys 0.004s
I20260927 23:30:48.134099 2808 raft_consensus.cc:348] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:48.135203 2808 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:48.136062 2808 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:48.139073 2808 ts_tablet_manager.cc:1428] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:30:48.143075 1598 tablet_copy_service.cc:342] P 0fe98f486bab4d0b9c421c145d83f1ce: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f662be9387d148199907fd0e049ba4be received from {username='slave'} at 127.0.0.1:56820
I20260927 23:30:48.143483 1598 tablet_copy_service.cc:433] P 0fe98f486bab4d0b9c421c145d83f1ce: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-f662be9387d148199907fd0e049ba4be on tablet f662be9387d148199907fd0e049ba4be with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:48.157904 1483 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:48.167591 1579 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.169744 2809 tablet_copy_client.cc:791] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 0 data blocks...
I20260927 23:30:48.170531 2809 tablet_copy_client.cc:655] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Starting download of 1 WAL segments...
I20260927 23:30:48.170413 2780 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:48.174748 2809 tablet_copy_client.cc:531] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:30:48.191666 1579 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.194064 2780 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:48.194955 1662 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:48.196497 2738 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:48.199671 2809 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap starting.
I20260927 23:30:48.237270 2809 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:30:48.238171 2809 tablet_bootstrap.cc:492] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Bootstrap complete.
I20260927 23:30:48.238986 2809 ts_tablet_manager.cc:1397] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.009s
I20260927 23:30:48.245296 2809 raft_consensus.cc:348] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:48.246493 2809 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Initialized, Role: LEARNER
I20260927 23:30:48.247370 2809 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:48.250084 2809 ts_tablet_manager.cc:1428] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260927 23:30:48.252148 1673 tablet_copy_service.cc:342] P 1b945905616b4989853b0a92bfbd6b98: Request end of tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-9c719671f35640549c1b944112b3fc03 received from {username='slave'} at 127.0.0.1:36962
I20260927 23:30:48.252540 1673 tablet_copy_service.cc:433] P 1b945905616b4989853b0a92bfbd6b98: ending tablet copy session 0ebfcbb9c0574725a9df6ed2bf33ff47-9c719671f35640549c1b944112b3fc03 on tablet 9c719671f35640549c1b944112b3fc03 with peer 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:48.282859 1730 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:48.287786 2738 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:48.291430 1656 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.294474 1729 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.308856 2839 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:48.309034 2775 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:48.382551 2732 raft_consensus.cc:1050] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:48.384866 2732 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:48.405052 2292 raft_consensus.cc:1203] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:48.410220 1730 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:48.412160 2292 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:48.416179 1580 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:48.417567 2711 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:48.419745 2732 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:48.421490 2711 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:48.427657 1579 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.429456 2838 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:48.442615 1586 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 23:30:48.463598 2732 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:48.478824 1663 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:48.480264 2775 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:48.489773 1734 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 23:30:48.499056 2293 raft_consensus.cc:1203] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 23:30:48.501672 2711 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:48.504719 2293 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Advancing to term 2
I20260927 23:30:48.528926 2732 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:48.548710 2292 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:48.574118 2293 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 23:30:48.576815 2849 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:48.598975 1580 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:48.614254 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 76663.4us
I20260927 23:30:48.771699 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 23:30:49.432617 1730 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.007s user 0.005s sys 0.017s
W20260927 23:30:49.434295 1730 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s user 0.006s sys 0.017s
I20260927 23:30:49.435390 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 974) took 1013 ms. Trace:
I20260927 23:30:49.436844 1730 rpcz_store.cc:276] 0927 23:30:48.421912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:48.423579 (+ 1667us) service_pool.cc:227] Handling call
0927 23:30:48.424149 (+ 570us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:30:48.424503 (+ 354us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:30:48.424536 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:30:48.425464 (+ 928us) log.cc:842] Serialized 540 byte log entry
0927 23:30:49.434860 (+1009396us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:30:49.434905 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:49.434988 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:30:49.435063 (+ 75us) raft_consensus.cc:1671] finished
0927 23:30:49.435085 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:30:49.435279 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1009100,"threads_started":1,"wal-append.queue_time_us":253507}
I20260927 23:30:49.667259 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 145) took 1249 ms. Trace:
I20260927 23:30:49.667649 1482 rpcz_store.cc:276] 0927 23:30:48.417501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:48.418304 (+ 803us) service_pool.cc:227] Handling call
0927 23:30:49.667172 (+1248868us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260927 23:30:49.681044 2859 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.681702 2859 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.692826 2859 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.734380 2856 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.734972 2856 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.737418 2856 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.737883 2856 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.743412 2863 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.746548 2859 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.747300 2859 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.750149 2859 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.743945 2863 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.760839 2863 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.764672 2864 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.768442 2853 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.768051 2864 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.768936 2853 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.770952 2864 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.771693 2853 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.773604 2859 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.774411 2859 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.775718 2864 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.777200 2859 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.776549 2864 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.777791 2866 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.778297 2866 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.744477 2856 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.780247 1730 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:49.780735 2856 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.779937 2864 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.781157 2853 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.781677 2864 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.782179 2868 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.782460 2864 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.782711 2868 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.781752 2853 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.785473 2864 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.786926 1663 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:49.787673 2868 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.787969 2856 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.788545 2856 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.790583 2856 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.791188 2853 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.792966 2863 raft_consensus.cc:1050] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
W20260927 23:30:49.794239 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 50185.1us
I20260927 23:30:49.794514 2867 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.795010 2867 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.795354 2863 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.797654 2867 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.810348 2867 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.810618 2868 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.810935 2867 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.811141 2868 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.812855 2868 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.813314 2853 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.813979 2853 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.819315 2853 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.820088 2859 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.821313 2859 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.822755 2868 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:49.823546 2859 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.816200 2866 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.823254 2868 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.832010 2868 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.820392 2867 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:49.840619 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 45698.5us
I20260927 23:30:49.863544 2864 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.865494 2864 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:49.868361 2864 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:30:49.861397 2859 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.879951 2859 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.882484 2862 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.884897 2871 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.886772 2859 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.887840 2864 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.890506 2868 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.892895 2870 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.898782 2861 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.892127 2871 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.884341 2862 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.903936 2859 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:49.910229 2857 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.922647 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.922134 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.922072 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.923794 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.927521 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.928194 1657 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:49.924975 2859 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.929049 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.920106 2855 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.914973 2861 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.910450 2838 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.931586 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:49.924892 2866 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.930285 2859 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.951370 2859 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.951052 2866 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:49.951560 2875 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:49.952298 2865 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.951885 2859 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.911548 2864 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.922466 2860 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.912828 2868 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.958052 2876 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.958297 2859 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.958771 2876 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.974187 2876 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.981299 2863 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.935467 2853 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.981880 2863 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.982026 2859 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.982535 2853 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.982816 2859 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.983820 2863 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.984431 2853 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.984913 2853 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.985361 2853 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.986653 2863 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.987216 2859 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:49.987689 2868 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.914126 2870 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.988222 2868 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.987906 2863 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.988689 2868 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.989571 2870 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.990036 2870 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.953500 2866 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.940645 2867 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.951931 2856 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.992358 2863 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.998577 2863 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:49.952953 2865 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.957775 2878 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.995048 2868 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:49.987727 2859 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.996287 2866 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.933483 2857 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:49.987707 2862 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:49.997412 2871 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.016701 1737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08bafa5142a9452db744d03134c85be6"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:49.999158 2863 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.019925 1737 raft_consensus.cc:595] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:49.931908 2855 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.020743 2877 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.021245 2874 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.005141 2876 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:30:50.019630 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 87731.4us
I20260927 23:30:49.952100 2875 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.935221 2838 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.013725 2868 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:49.993672 2870 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:49.993571 2867 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.036394 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.003171 2856 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.039623 1729 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.015938 2866 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.083189 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.084488 2886 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:50.018373 2859 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.035481 2862 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:49.980219 2860 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.091588 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.095760 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.096452 1656 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.096740 2884 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.036513 2877 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.100811 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.097430 2884 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.105188 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.076251 2874 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.039927 2876 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.018949 2878 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:50.116994 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 87042.2us
I20260927 23:30:50.043951 2868 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.102260 2853 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.111249 2871 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.098625 2862 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.140863 2864 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.138797 2854 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.142416 2872 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:50.161279 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.163511 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:50.164932 2804 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260927 23:30:50.165067 2889 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.165400 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.158890 2880 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.175525 1587 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:50.164960 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.182106 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.182632 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.182835 2884 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.106380 2886 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.183238 1587 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:50.183370 2884 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.163718 2877 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.184525 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.149667 2871 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.148347 2869 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.179s
I20260927 23:30:50.208777 2860 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.209264 2860 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.209471 2871 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.209632 2860 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.211747 2856 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.212211 2856 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.212594 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.213135 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.213641 2879 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.213923 2294 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.177536 2863 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.212545 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.212709 2892 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:50.183925 2884 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.186784 2886 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.214151 2879 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.178184 2865 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.207479 2875 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.183420 1585 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 23:30:50.206743 2880 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.287422 2891 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.207755 2855 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.211190 2838 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.210836 2870 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.212126 2874 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.211407 2857 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.296922 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.299768 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.165633 2889 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.214588 2882 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.180804 2854 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.303966 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.304937 2294 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.290217 2859 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.299458 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.305678 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.212615 2888 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.306975 1660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:50.230094 2866 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.281862 2871 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:50.318660 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.304982 1656 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
W20260927 23:30:50.314957 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 85945us
I20260927 23:30:50.306232 2878 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.304545 2865 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.302282 2885 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.325384 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.317723 1660 raft_consensus.cc:595] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:50.318908 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.212773 2856 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.302743 2897 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.332067 2865 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.281659 2887 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.226119 2893 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:50.317526 2876 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.324354 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.331115 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.328459 1588 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:50.288043 2891 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:50.335920 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.331795 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.311985 2874 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.181001 2872 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.371793 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.376447 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.377059 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.281634 2892 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.340075 2862 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.378521 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.334257 2897 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.323127 2871 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.391206 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:50.471763 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 93312.5us
I20260927 23:30:50.378252 2875 raft_consensus.cc:1050] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
W20260927 23:30:50.472399 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 95483.4us
I20260927 23:30:50.472867 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.421737 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:50.479414 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 170592us
I20260927 23:30:50.445153 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.481739 1728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:50.465689 2860 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.471150 2880 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.475747 2869 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.483223 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:50.372202 2893 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.487568 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:50.478186 2868 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.482360 1728 raft_consensus.cc:595] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:50.482462 2876 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.481777 2865 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.388476 2874 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.491114 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.487145 2884 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.325230 2888 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.380738 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.487251 2869 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.319303 2882 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.497264 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.492807 2876 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.491906 2879 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.502687 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.500976 2860 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.503607 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.503294 2900 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.498090 2884 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.498310 2869 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.512166 2892 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.512987 2856 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.543396 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.507704 2876 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.387609 2885 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.506950 2877 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.513346 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260927 23:30:50.377223 2887 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:50.607306 1734 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:50.561888 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.507274 2860 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.614257 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.592435 2889 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.511016 2884 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.513058 2871 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.610824 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.566556 2861 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.567515 2891 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:50.618345 1758 debug-util.cc:398] Leaking SignalData structure 0x7b08001d8ec0 after lost signal to thread 399
I20260927 23:30:50.562420 2872 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.513098 2854 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.562608 2870 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260927 23:30:50.616654 2860 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.513226 2879 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.622468 2884 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.614038 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.632149 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.565929 2900 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.626215 2893 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.614146 2877 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.640254 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.643347 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.634424 2874 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.637917 2879 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.639302 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.640851 2882 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.649583 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.642863 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.637198 2875 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.650269 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:50.651048 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 170227us
I20260927 23:30:50.652128 2879 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.656991 2860 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:50.645875 1758 debug-util.cc:398] Leaking SignalData structure 0x7b080019f160 after lost signal to thread 1520
I20260927 23:30:50.639750 2897 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:50.662263 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 51131.5us
I20260927 23:30:50.665560 2258 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:50.648706 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.666683 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.670682 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.673321 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.678758 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.680424 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.680599 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.680454 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.680929 2885 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.681082 1662 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.681264 1657 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.679544 1663 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:50.668431 2888 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.673933 1733 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:50.681567 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.685432 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.685642 2899 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.686566 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.689751 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.690208 2258 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.668483 2879 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 23:30:50.681849 1758 debug-util.cc:398] Leaking SignalData structure 0x7b080020da80 after lost signal to thread 1612
I20260927 23:30:50.693471 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.694482 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.695961 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.696887 1551 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.697335 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.698751 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.699883 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.699985 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.700369 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.702886 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.703426 1734 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
W20260927 23:30:50.694711 1758 debug-util.cc:398] Leaking SignalData structure 0x7b0800357d80 after lost signal to thread 1613
I20260927 23:30:50.702030 2856 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.705859 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.693912 2879 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:50.708113 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.695046 2887 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:50.708928 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.708012 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.711576 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.716960 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.718353 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.719859 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.720155 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.720491 2291 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:50.721086 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.711344 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.722067 2867 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.722225 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.722537 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.709064 2907 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.723172 2907 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.723333 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.723558 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.723979 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.710747 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.712903 2908 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.727957 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.710070 2259 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:50.709066 2856 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.704732 2900 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.739630 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:50.711105 1758 debug-util.cc:398] Leaking SignalData structure 0x7b0800361b80 after lost signal to thread 1687
I20260927 23:30:50.719796 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.709671 2879 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.737161 2293 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:50.710011 2901 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.748322 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:50.748610 2901 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.749019 2901 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.731120 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.709983 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.742820 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.743242 2860 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.723563 2907 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.744704 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:50.745097 2904 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260927 23:30:50.746474 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.739120 2856 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.760092 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:50.749902 1758 debug-util.cc:398] Leaking SignalData structure 0x7b08000ead20 after lost signal to thread 1688
I20260927 23:30:50.752818 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.731688 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.756954 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 985) took 2212 ms. Trace:
I20260927 23:30:50.757357 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.757771 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:50.762773 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.766497 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.768715 1579 rpcz_store.cc:276] 0927 23:30:48.544644 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:49.987909 (+1443265us) service_pool.cc:227] Handling call
0927 23:30:49.988198 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:50.090364 (+102166us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:30:50.090391 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:30:50.090426 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:30:50.090452 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:50.756779 (+666327us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54003b47d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:30:50.756818 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:30:50.756907 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234528128}
I20260927 23:30:50.762962 1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48984 (request call id 240) took 2199 ms. Trace:
I20260927 23:30:50.750826 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.762668 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.771711 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.753561 2908 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.773629 1657 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:50.774762 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.772981 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.770043 1484 rpcz_store.cc:276] 0927 23:30:48.558732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:48.567158 (+ 8426us) service_pool.cc:227] Handling call
0927 23:30:50.757674 (+2190516us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1656,"reported_tablets":3}
I20260927 23:30:50.729202 2869 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.772621 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:50.776623 1486 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48998 (request call id 229) took 2206 ms. Trace:
W20260927 23:30:50.762995 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 58776.6us
I20260927 23:30:50.779196 1731 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 23:30:50.782253 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.782804 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.784807 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.780081 1486 rpcz_store.cc:276] 0927 23:30:48.570402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:48.574476 (+ 4074us) service_pool.cc:227] Handling call
0927 23:30:50.776487 (+2202011us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166830,"reported_tablets":1}
I20260927 23:30:50.787333 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.793229 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.794685 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.798530 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.801997 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.802134 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.806551 2292 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:50.806593 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.808591 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.810633 1485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48976 (request call id 242) took 2210 ms. Trace:
I20260927 23:30:50.811106 1485 rpcz_store.cc:276] 0927 23:30:48.599786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:48.599907 (+ 121us) service_pool.cc:227] Handling call
0927 23:30:50.810575 (+2210668us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5499,"reported_tablets":2}
I20260927 23:30:50.785051 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:50.813200 2289 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.813639 2869 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.785566 2838 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.795233 2899 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.816573 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:50.817375 1657 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.816612 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.761484 2876 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.813668 1482 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:50.819747 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:30:50.808873 1758 debug-util.cc:398] Leaking SignalData structure 0x7b080031e740 after lost signal to thread 1761
I20260927 23:30:50.830703 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.831244 1700 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:50.807312 2877 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.834823 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.837234 1700 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:50.837524 2292 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:50.837433 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.814776 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:50.823750 2899 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.835659 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.838304 1700 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:50.838845 1628 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:30:50.841749 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 100382us
I20260927 23:30:50.842669 1628 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.844389 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.836347 1583 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:50.840116 2880 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.849563 2888 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.850001 2888 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.851439 2888 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.834933 2892 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:50.847762 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.846303 2865 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:50.839030 2905 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 23:30:50.840508 1758 debug-util.cc:398] Leaking SignalData structure 0x7b08002280c0 after lost signal to thread 2318
I20260927 23:30:50.856248 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.840698 2899 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.860551 2906 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.861083 2884 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.847620 2893 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.875699 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.855738 2880 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.793974 1732 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:50.855441 2892 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:50.858886 2874 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.887251 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:50.864064 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.887887 1659 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.859158 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.840701 2898 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.862270 2879 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:50.876981 2893 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.878634 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.882130 1657 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:50.884035 2864 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:50.884593 2876 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:50.881166 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:50.885751 2884 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:50.858428 2905 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:30:50.916326 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 78090.8us
I20260927 23:30:50.897039 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.899147 1552 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.900801 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.901993 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.903307 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.904754 2893 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.901788 2873 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:50.908856 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.927053 1660 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:50.928006 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.909080 2900 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:50.879846 2865 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260927 23:30:50.909464 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:50.910853 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:50.930532 2294 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.905584 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.931027 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.931649 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.919023 2291 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:50.907500 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:50.894771 2892 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:50.932476 1552 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.935447 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.909083 2864 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:50.940279 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:50.917670 2905 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.948710 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:50.928965 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 56823.1us
I20260927 23:30:50.953382 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.931883 2900 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:50.934629 2867 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.955278 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.956054 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:50.956766 2292 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 23:30:50.943202 2894 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:30:50.957998 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 69534.6us
I20260927 23:30:50.942840 2864 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:50.951699 2838 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:50.951905 2898 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.958881 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.951313 2884 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:50.959022 2876 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.967036 2875 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.982661 2900 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.982568 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.953028 2874 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.985690 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:50.954413 2289 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:50.996925 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.997641 2292 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:50.991110 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:50.999557 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.001144 2896 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:50.983878 2838 raft_consensus.cc:686] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:51.010462 2293 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.007531 2907 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:50.990999 2290 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:51.011844 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.012317 2294 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:51.021819 2903 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:51.022022 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.023543 2871 raft_consensus.cc:979] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:51.023976 2871 raft_consensus.cc:1067] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:51.030069 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.042218 2291 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.041944 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:51.043342 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45711.4us
I20260927 23:30:51.044516 1627 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.048755 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.048903 2865 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.049471 2907 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.018303 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:51.054061 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.020239 2879 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.071255 2917 raft_consensus.cc:979] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: : Instructing follower 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:51.029557 2903 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:51.073706 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.076362 2903 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.077353 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:30:51.078423 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 48476.1us
I20260927 23:30:51.079385 1627 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.079869 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:51.045110 2876 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:51.074679 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:51.081955 2915 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:51.083036 2915 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:51.083263 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.083442 2915 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:51.083745 1659 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:51.084383 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.024497 2860 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260927 23:30:51.085700 2918 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.085961 1583 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:51.083894 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:51.086270 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.087373 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.093134 1627 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.094480 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.092208 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:51.097010 2259 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.098050 2865 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.098629 2867 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.099988 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.084892 2896 raft_consensus.cc:1067] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Signalling peer 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:51.101006 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.095649 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.090170 2876 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:51.082995 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.111152 1728 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:51.112934 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.113111 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.114933 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.107447 2872 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.109638 2865 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260927 23:30:51.094197 2917 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:51.124617 1703 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.127444 1703 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:51.132735 2911 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260927 23:30:51.111459 1583 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.082360 2838 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.134824 1703 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:51.122740 2917 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:51.150077 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.134516 2871 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:51.136438 2880 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.156383 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.174129 1663 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.176075 1703 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.154922 2871 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:51.155391 2917 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.188601 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.195124 2258 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.196127 2258 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.171820 2856 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.188601 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.177146 2897 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:51.206063 1629 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.206296 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.183902 2880 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:51.212749 1662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08bafa5142a9452db744d03134c85be6"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46200
I20260927 23:30:51.213490 1662 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:51.213894 1662 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.193797 2871 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.166220 2899 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.217063 1580 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.218670 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.226483 1700 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.227752 1700 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.202581 2889 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.198987 2879 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.203028 2869 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.205539 2906 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:51.226656 2881 consensus_queue.cc:1046] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260927 23:30:51.191252 2914 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260927 23:30:51.208582 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:51.210503 2880 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:51.215178 2872 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.266261 1581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:46518
I20260927 23:30:51.239233 1662 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.186527 2901 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.181382 2876 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.193662 2861 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260927 23:30:51.227334 2913 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.313509 1581 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:51.318661 1581 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.314949 1662 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.265887 2887 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.302016 2868 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.215999 2893 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.312392 2880 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.346758 1586 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:51.358361 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 147246us
I20260927 23:30:51.216290 2888 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.360942 2856 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.361304 2888 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.359239 2897 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 23:30:51.362391 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:51.363180 2923 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260927 23:30:51.371348 2899 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.370782 2864 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.372877 1655 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.373942 2914 raft_consensus.cc:3032] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.374337 2914 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:51.375239 2914 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.376724 2914 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.406476 2888 raft_consensus.cc:1050] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: attempting to promote NON_VOTER 0ebfcbb9c0574725a9df6ed2bf33ff47 to VOTER
I20260927 23:30:51.409240 1656 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.410501 2918 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:51.411557 2918 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:51.348707 2906 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:51.411365 2888 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.434016 2929 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 23:30:51.455292 2901 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.345602 2922 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.134s
I20260927 23:30:51.456219 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.457659 2922 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.458097 2922 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:51.410743 2864 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:51.511600 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 104173us
I20260927 23:30:51.512460 1629 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:51.512022 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 319498us
W20260927 23:30:51.481287 2892 consensus_meta.cc:287] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.531s user 0.011s sys 0.007s
W20260927 23:30:51.477087 2905 consensus_meta.cc:287] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.523s user 0.013s sys 0.003s
I20260927 23:30:51.522708 1587 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.483163 2893 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.535599 2917 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.535447 2894 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:51.536199 1581 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.543968 2927 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.544730 2922 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.544350 2288 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:51.545986 2922 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.472954 2915 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.544368 2927 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:51.551519 1659 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:51.549173 1581 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:51.550209 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:51.558128 2291 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.563133 2934 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:51.563889 2934 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.547396 2865 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.525480 2892 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.547830 2917 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.594817 2858 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:51.526988 2905 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.552752 2915 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:51.638852 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 398674us
I20260927 23:30:51.639902 1700 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.642608 1700 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.610626 2913 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260927 23:30:51.624682 2938 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.481928 2906 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.650655 2938 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:51.650997 2931 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:51.655260 2294 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:51.593637 2927 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.657606 2927 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.658095 2860 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:30:51.641386 2934 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:51.663719 2924 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.305s user 0.011s sys 0.016s
W20260927 23:30:51.665182 2933 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260927 23:30:51.595561 2858 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: true } }
I20260927 23:30:51.671702 2868 raft_consensus.cc:3032] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.672180 2868 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:51.674595 2868 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.677438 2868 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.667342 2934 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.682173 2291 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:51.689538 2858 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.694808 2874 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:51.697752 2874 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.698086 2880 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.699126 1554 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:51.699659 2874 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.700197 2901 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:51.700687 2901 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:51.688078 2892 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:51.701221 2905 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:51.666869 2931 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.690438 2938 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.708534 1734 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:51.726140 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.021s sys 0.022s
W20260927 23:30:51.726567 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.023s sys 0.022s
W20260927 23:30:51.726836 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 503131us
I20260927 23:30:51.713528 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.715636 2880 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:51.734856 1628 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.743866 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1124) took 1106 ms. Trace:
I20260927 23:30:51.747328 1586 rpcz_store.cc:276] 0927 23:30:50.637647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:50.725693 (+ 88046us) service_pool.cc:227] Handling call
0927 23:30:50.725923 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:51.355055 (+629132us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:51.743610 (+388555us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540038f4d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:30:51.743797 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1427405696}
I20260927 23:30:51.757258 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.758711 1628 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.760036 1628 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.763018 1729 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:51.763921 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 146) took 2009 ms. Trace:
I20260927 23:30:51.727626 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.748500 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.770780 2932 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.780242 2931 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:51.781683 1659 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.771981 2894 consensus_queue.cc:1046] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260927 23:30:51.779310 2892 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.786559 1628 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:51.789637 2871 consensus_meta.cc:287] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.507s user 0.002s sys 0.007s
I20260927 23:30:51.790503 2871 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.792348 2871 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:51.767385 2929 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.768632 2918 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.768815 2927 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.795610 1628 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.782735 2891 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.793051 2932 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.814996 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:51.821805 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 179234us
I20260927 23:30:51.802214 2938 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.764389 1482 rpcz_store.cc:276] 0927 23:30:49.754494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:49.755273 (+ 779us) service_pool.cc:227] Handling call
0927 23:30:51.763856 (+2008583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:30:50.951234 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:30:51.020434 (+ 69200us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:30:51.025496 (+ 5062us) write_op.cc:435] Acquired schema lock
0927 23:30:51.028284 (+ 2788us) tablet.cc:660] Decoding operations
0927 23:30:51.071757 (+ 43473us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:30:51.071839 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0927 23:30:51.071879 (+ 40us) tablet.cc:683] Acquiring locks for 1 operations
0927 23:30:51.072070 (+ 191us) tablet.cc:699] Row locks acquired
0927 23:30:51.072089 (+ 19us) write_op.cc:260] PREPARE: finished
0927 23:30:51.072231 (+ 142us) write_op.cc:270] Start()
0927 23:30:51.072339 (+ 108us) write_op.cc:276] Timestamp: P: 1790551851072190 usec, L: 0
0927 23:30:51.072361 (+ 22us) op_driver.cc:331] REPLICATION: starting
0927 23:30:51.074050 (+ 1689us) log.cc:842] Serialized 587 byte log entry
0927 23:30:51.281593 (+207543us) op_driver.cc:444] REPLICATION: finished
0927 23:30:51.527692 (+246099us) write_op.cc:301] APPLY: starting
0927 23:30:51.527799 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0927 23:30:51.527899 (+ 100us) tablet.cc:1368] finished BulkCheckPresence
0927 23:30:51.528024 (+ 125us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:30:51.555613 (+ 27589us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:30:51.580377 (+ 24764us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0927 23:30:51.596549 (+ 16172us) write_op.cc:312] APPLY: finished
0927 23:30:51.599939 (+ 3390us) log.cc:842] Serialized 23 byte log entry
0927 23:30:51.600177 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:51.600283 (+ 106us) write_op.cc:454] Released schema lock
0927 23:30:51.600341 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0927 23:30:51.600709 (+ 368us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:51.600747 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":894014,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":215327,"apply.run_cpu_time_us":2616,"apply.run_wall_time_us":73334,"num_ops":1,"prepare.queue_time_us":85355,"prepare.run_cpu_time_us":7060,"prepare.run_wall_time_us":171616,"raft.queue_time_us":37898,"raft.run_cpu_time_us":7295,"raft.run_wall_time_us":309072,"replication_time_us":163573,"thread_start_us":258547,"threads_started":4,"wal-append.queue_time_us":78126}]]}
I20260927 23:30:51.826239 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.807389 2891 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:51.802579 2929 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.829629 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.823841 1700 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.829968 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.809190 2934 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.767290 2901 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.821551 2922 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.849217 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.855826 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.856395 1587 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.866811 2910 raft_consensus.cc:3032] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:51.850594 2943 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.867194 2910 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:51.867936 2910 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.868419 1734 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
W20260927 23:30:51.870560 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 46811.6us
I20260927 23:30:51.871254 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.858426 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.875061 1587 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
W20260927 23:30:51.865978 2926 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.508s user 0.003s sys 0.000s
I20260927 23:30:51.875986 1700 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.858426 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.882884 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.884294 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:51.888207 1700 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.888978 2861 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.890264 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.892342 1700 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.891191 1583 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:51.877823 2943 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.876708 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:51.896375 1700 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.891007 2931 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.886606 2910 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:51.890787 2918 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:51.895682 2946 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.897435 2861 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:30:51.898841 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 350032us
I20260927 23:30:51.927026 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:51.919603 2922 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:30:51.929845 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 417296us
I20260927 23:30:51.933302 2947 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:51.892453 2891 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.944932 2891 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.929277 1583 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.956265 2293 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.935504 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:51.939214 1700 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.963550 1700 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.964279 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:51.965153 1700 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.966274 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:51.966800 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.968791 1629 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:51.969488 2922 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.969836 1629 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:51.970863 1629 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.972076 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:51.973559 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.973785 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:51.976205 2931 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:51.976796 2918 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:51.977620 2918 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:51.978513 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:51.979270 1656 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:51.981671 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.982627 1700 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.983512 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.986016 1657 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.987865 1663 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:51.988724 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.989153 1662 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.989375 1662 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:51.994262 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:51.994592 1700 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:51.996189 2872 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:51.996603 1662 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:51.997679 2872 raft_consensus.cc:3032] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:51.997780 2897 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.998489 2944 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:51.980043 2946 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.998081 2872 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:52.009209 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:52.011193 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:52.016222 1662 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:51.998857 2913 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260927 23:30:52.022770 1587 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:52.010047 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:51.987879 2871 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:51.990187 2916 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.032292 1661 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:51.992285 2915 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:51.994522 2893 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.037793 1662 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:52.002102 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:52.043437 1654 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.044817 2289 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.044826 1733 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.047128 2948 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:51.994594 2947 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:52.056465 2944 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.056931 2944 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:51.984567 2861 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260927 23:30:52.058727 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.059249 2861 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.059155 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.059827 2861 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:52.060020 1588 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.060738 2861 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.061345 2946 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:52.065136 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.000880 2914 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.059355 1733 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.072316 1733 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:30:52.050259 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57971.6us
I20260927 23:30:52.067709 1485 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:30:51.984853 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 95884.6us
I20260927 23:30:52.073827 2258 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.073210 1737 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:52.062103 2861 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.076417 1737 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:52.073204 1733 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.077122 1733 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.073794 1627 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.075984 1485 catalog_manager.cc:5495] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.077905 1736 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.074723 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.057828 2944 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.081005 2889 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.076937 2901 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260927 23:30:52.082463 2872 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.034308 2896 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260927 23:30:52.086980 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.087165 2872 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.079331 1485 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.089345 1588 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.092759 1733 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:52.092993 1657 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.093809 1730 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.093742 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:52.098927 1730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08bafa5142a9452db744d03134c85be6"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:52.103166 1580 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:52.104918 1703 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:52.106033 1703 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:52.079255 1736 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.072237 2914 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:52.121670 1703 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:52.090049 2938 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.123497 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:52.125036 1703 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:52.126184 2893 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260927 23:30:52.126791 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.125869 2891 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:52.124461 2947 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260927 23:30:52.121229 2868 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:52.106984 2952 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:52.102797 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.120106 2881 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.142911 2918 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260927 23:30:52.149535 2947 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.133895 2896 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260927 23:30:52.151429 2947 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:52.144412 2915 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.148691 1735 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.149963 1730 raft_consensus.cc:595] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:52.152318 2947 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.158914 2938 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:52.159170 1702 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:52.159976 2915 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.118676 2931 consensus_queue.cc:1046] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260927 23:30:52.160806 2915 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:52.157759 2896 raft_consensus.cc:3032] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.161315 2896 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:52.162158 2947 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.162083 2896 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.159999 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:52.139786 2897 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.157759 1730 raft_consensus.cc:603] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:30:52.161725 2915 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.169965 2914 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.177481 2914 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.175417 2896 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.179113 2915 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.182381 1551 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.185405 2288 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 LEARNER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 23:30:52.187991 1661 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.183013 2257 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.191164 2292 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 23:30:52.191781 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:52.173774 2892 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:52.194727 2838 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.200969 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:52.197249 2952 consensus_queue.cc:1046] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20260927 23:30:52.202644 1662 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.203795 1629 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.205987 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.215140 2899 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:30:52.215492 2953 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260927 23:30:52.193468 2854 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.209052 2941 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:52.213975 2838 raft_consensus.cc:3032] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:52.203157 2892 raft_consensus.cc:686] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:52.235952 2838 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:52.240579 2838 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.241734 2838 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:52.222910 2952 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260927 23:30:52.216329 2888 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260927 23:30:52.223587 2918 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260927 23:30:52.224545 2874 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260927 23:30:52.247885 2892 consensus_queue.cc:237] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:52.284984 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 48692.1us
I20260927 23:30:52.289760 2903 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:52.290916 2903 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.291249 2903 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:52.304576 2891 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:52.305059 2891 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:52.309150 2944 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.310285 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.311326 2901 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.326352 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.291952 2903 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.305823 2891 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.309149 1732 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:52.330291 2874 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.363391 1736 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:52.364197 1736 raft_consensus.cc:595] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:30:52.411223 2919 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260927 23:30:52.358387 2910 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260927 23:30:52.430671 2903 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.433939 1587 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:52.483107 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 140102us
I20260927 23:30:52.483839 1551 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.488710 1662 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:52.489306 1662 raft_consensus.cc:595] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:52.489637 1662 raft_consensus.cc:603] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:30:52.490559 1551 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.515779 1662 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:52.518049 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260927 23:30:52.519639 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:52.520262 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:52.517040 2906 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:52.535141 2289 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.536972 2860 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:30:52.513232 2874 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:30:52.566439 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50383.3us
I20260927 23:30:52.572365 2287 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.555539 2889 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:52.587126 1627 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:52.578856 1735 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:52.549341 2854 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 23:30:52.592842 2872 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:52.603749 2944 raft_consensus.cc:979] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:52.608917 2944 raft_consensus.cc:1067] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 0ebfcbb9c0574725a9df6ed2bf33ff47 to start an election
I20260927 23:30:52.586233 1728 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:52.589802 2889 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.594805 1655 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:52.624980 2291 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
from {username='slave'} at 127.0.0.1:51538
I20260927 23:30:52.625535 2291 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:52.625828 2291 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.635478 2891 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:52.649995 2969 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 23:30:52.610492 2860 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260927 23:30:52.573861 2938 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.621289 2872 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:52.678911 2291 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.716702 2291 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:52.727118 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:52.727818 1580 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.732986 1588 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.745085 1580 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:52.755664 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:52.756469 1663 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:52.759517 1627 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:52.761471 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:52.767122 1585 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:52.770567 2872 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:52.771329 2972 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260927 23:30:52.773620 2956 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:52.773998 2956 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:52.774341 2956 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:52.781793 2889 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:52.783879 2901 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:52.784440 2901 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.782287 2259 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:30:52.743677 2888 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:52.791745 2889 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:52.791671 2888 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.792280 2889 raft_consensus.cc:686] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:30:52.792510 2967 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.181s user 0.002s sys 0.000s
I20260927 23:30:52.793147 2889 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.793748 2888 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:52.796404 2915 consensus_meta.cc:287] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.617s user 0.006s sys 0.004s
I20260927 23:30:52.801817 1730 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 23:30:52.834667 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 75256.2us
W20260927 23:30:52.850584 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 43482.6us
I20260927 23:30:52.844168 2929 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:52.851708 2929 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:52.852525 2929 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.853266 2941 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:52.855191 1585 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:52.857509 2897 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:30:52.859156 2901 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:52.860950 1663 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:52.866276 1662 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:52.877228 1588 raft_consensus.cc:1226] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:52.883525 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:52.797431 2873 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.888845 1663 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:30:52.901764 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:52.902753 1736 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:52.889008 1657 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:52.907959 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 66100.5us
I20260927 23:30:52.910190 1551 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:52.918493 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:52.919201 2292 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:52.907397 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:52.939005 1734 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:52.944351 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:52.900177 2946 consensus_queue.cc:1046] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 23:30:52.917140 2903 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:52.914356 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:52.959128 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:52.933301 2854 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 23:30:52.914037 2910 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260927 23:30:52.911801 2880 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260927 23:30:52.970175 1731 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:52.974373 2903 raft_consensus.cc:686] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:52.985229 2903 consensus_queue.cc:237] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:52.961328 2908 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:52.988744 2908 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:52.990705 2908 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:52.995343 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:52.914312 2979 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260927 23:30:53.014976 1735 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:53.032372 1736 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:53.045292 1728 raft_consensus.cc:3032] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:53.045763 1728 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:53.046542 1728 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:53.048677 2978 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260927 23:30:53.032649 2968 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260927 23:30:52.984253 2956 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:53.058100 2956 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:53.067077 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 151316us
I20260927 23:30:53.097358 1551 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:53.097975 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:53.104521 1737 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:53.114647 2910 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:30:53.152380 2986 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260927 23:30:53.110241 2938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46178 (request call id 868) took 1302 ms. Trace:
I20260927 23:30:53.172106 1728 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:53.172519 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 53438.7us
I20260927 23:30:53.173179 2973 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:53.173683 2973 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.176296 2973 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:53.218425 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:53.219451 1729 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:53.231503 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:53.243582 2976 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:53.244113 2976 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:53.244518 2976 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:53.256975 1729 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:53.258311 2915 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:53.166268 2938 rpcz_store.cc:276] 0927 23:30:51.779924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:51.780059 (+ 135us) service_pool.cc:227] Handling call
0927 23:30:51.784742 (+ 4683us) log.cc:842] Serialized 480 byte log entry
0927 23:30:52.473307 (+688565us) spinlock_profiling.cc:243] Waited 944 us on lock 0x7b5400056d18. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a4437c 00007f9750a46e80 00007f9750a46e08 00007f9750a46d88 00007f9750a46d43 00007f9750a4600c 00007f9756a78206 00007f9756a70808 00007f97506fdeb1 00007f97506f12aa
0927 23:30:53.067025 (+593718us) log.cc:842] Serialized 14 byte log entry
0927 23:30:53.082232 (+ 15207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78018,"raft.queue_time_us":107289,"raft.run_cpu_time_us":982,"raft.run_wall_time_us":29326,"spinlock_wait_cycles":2170240,"thread_start_us":165958,"threads_started":1,"wal-append.queue_time_us":318189}
I20260927 23:30:53.284224 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5ddb51781fec46b194174de5f4406def with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260927 23:30:53.306725 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 355329us
I20260927 23:30:53.318775 2289 raft_consensus.cc:1226] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:53.381562 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 459893us
I20260927 23:30:53.313285 2976 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:53.399518 2291 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:53.406817 1581 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:53.405752 2294 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:53.417019 2976 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:53.436069 2893 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:53.441097 2893 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.443068 2893 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:53.445166 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:53.450873 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:53.451735 1733 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:53.454569 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:53.479187 1734 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:53.479628 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:53.481436 1736 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:53.494661 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:53.495265 1660 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:53.495132 1581 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:53.507830 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:53.509907 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:53.510682 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:53.511227 2294 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:53.520612 1628 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:53.495433 2984 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:53.515218 2944 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:53.545157 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 53635us
I20260927 23:30:53.551775 1734 raft_consensus.cc:1261] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:53.694031 2294 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:53.547667 1728 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:53.544793 2952 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 23:30:53.706463 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 185904us
I20260927 23:30:53.647004 1733 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:53.633281 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 23:30:53.714143 2945 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
W20260927 23:30:53.691920 3005 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20260927 23:30:53.722958 1629 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:30:53.723059 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 147664us
I20260927 23:30:53.736874 1735 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:53.737198 1588 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:53.743188 1660 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
W20260927 23:30:53.783252 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 60356.2us
I20260927 23:30:53.736363 2998 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:53.550985 2873 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.234s
I20260927 23:30:53.727115 2861 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:53.748087 2891 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:53.747653 2915 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:53.554343 2944 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:53.848858 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 132232us
I20260927 23:30:53.715730 2952 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:53.785156 2871 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:53.720868 2897 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:53.786684 2916 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:53.800362 2893 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:53.730195 2854 consensus_queue.cc:1035] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.117s
I20260927 23:30:53.915810 2952 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:53.558195 2984 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.846465 2998 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.890552 2948 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260927 23:30:53.855325 2861 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.885030 2891 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.916374 2897 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.890388 2915 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:53.892154 2871 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.113091 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:54.114791 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 227328us
I20260927 23:30:54.132706 2981 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.133111 3031 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:54.133493 2891 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.133651 3031 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:54.134394 2915 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.134804 2891 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.135330 2891 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.137373 2891 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.137856 2891 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.139225 2871 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.139468 2998 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.141835 3033 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.142213 2861 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.101554 2984 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.095955 2944 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.053995 2893 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:54.143735 3031 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.133528 2981 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.139220 2897 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.138275 2891 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.152769 3033 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.178882 2981 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.172654 3035 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.183280 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.179734 2952 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.186566 2291 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:54.186367 3035 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.173805 2893 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.188313 3033 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.189308 3035 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.212212 2952 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.225173 2935 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.225962 2935 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.228358 2935 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.360045 2945 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.403455 2945 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.417747 2891 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.432325 2945 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.460153 2981 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.489605 2981 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:54.496258 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 278669us
I20260927 23:30:54.497308 2935 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.499681 3024 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.500214 3024 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.500856 2956 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:54.501441 2956 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.503046 3024 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.498090 3038 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.503952 3038 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.505131 2956 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.507532 3038 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.509202 3038 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:54.497910 2935 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.510475 3038 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.512305 3038 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.513236 2935 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.565423 2935 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.566166 2935 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.570047 2935 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.577179 2981 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.715593 2935 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.716360 2935 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.719866 2935 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.823498 3038 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.824910 3038 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.830451 3038 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.886109 2945 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.902038 2945 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.900576 2956 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.903064 2956 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.905332 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.905622 2956 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.906029 1728 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:54.906157 2956 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.907529 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:54.908216 1733 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:54.907792 2956 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.909165 2289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 294) took 1772 ms. Trace:
I20260927 23:30:54.909564 2289 rpcz_store.cc:276] 0927 23:30:53.136909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:53.141313 (+ 4404us) service_pool.cc:227] Handling call
0927 23:30:53.146875 (+ 5562us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:53.374966 (+228091us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:54.908908 (+1533942us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54006ce154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:30:54.909092 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":373789568}
I20260927 23:30:54.910523 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.911645 2956 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.911864 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.912204 2956 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.912436 1733 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:54.912817 2956 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.913868 1733 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:30:54.898979 2981 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.915068 2981 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.917725 2981 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.917614 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.917936 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.918524 1736 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:54.920069 2945 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.914057 3032 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.920807 3032 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.922384 2956 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.917495 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.923158 1730 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:54.923934 3032 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.929839 3045 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.930493 3045 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.930940 3032 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.931511 3032 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:54.934852 3032 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:54.936285 3045 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:54.936563 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:54.937222 1730 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:54.948426 3032 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:54.948976 3032 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.950805 3038 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:54.951365 3038 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.953902 3038 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.955718 3038 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:54.956380 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.956655 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.957072 2289 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:54.957254 2287 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:54.957175 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.958045 2290 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:54.958287 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.958629 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.958789 2258 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:54.959049 2294 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:54.959172 2289 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:54.956218 3038 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:54.961529 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.962677 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:54.962872 3038 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.963280 2293 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:54.964119 2956 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:30:54.964468 2956 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:54.964447 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:54.964885 2956 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:54.966037 3032 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:54.971109 2981 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:54.971582 2981 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:54.971974 2981 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:54.974608 3045 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260927 23:30:54.976395 1553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.418s user 0.090s sys 0.194s
W20260927 23:30:54.976719 1553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.419s user 0.090s sys 0.194s
W20260927 23:30:55.607041 1661 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.761s user 0.004s sys 0.040s
W20260927 23:30:55.607589 1661 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.805s user 0.004s sys 0.041s
I20260927 23:30:55.608345 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1352) took 2133 ms. Trace:
W20260927 23:30:55.609112 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.810s user 0.015s sys 0.017s
W20260927 23:30:55.609555 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.810s user 0.015s sys 0.017s
I20260927 23:30:55.608686 1661 rpcz_store.cc:276] 0927 23:30:53.474968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:53.478499 (+ 3531us) service_pool.cc:227] Handling call
0927 23:30:53.546669 (+ 68170us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:30:53.711945 (+165276us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:30:53.711986 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:30:53.713992 (+ 2006us) log.cc:842] Serialized 26 byte log entry
0927 23:30:55.607972 (+1893980us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:30:55.608013 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:55.608088 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:30:55.608134 (+ 46us) raft_consensus.cc:1671] finished
0927 23:30:55.608144 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:30:55.608271 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17263,"raft.queue_time_us":166896,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":1805119,"threads_started":1,"wal-append.queue_time_us":361886,"wal-append.run_cpu_time_us":2197,"wal-append.run_wall_time_us":1033827}
W20260927 23:30:55.658272 1626 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.856s user 0.001s sys 0.035s
W20260927 23:30:55.659327 1626 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.862s user 0.001s sys 0.035s
W20260927 23:30:55.791888 2981 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.819s user 0.015s sys 0.006s
I20260927 23:30:55.792358 2981 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.796674 2981 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:55.798482 3020 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.799321 3020 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.802757 3020 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.828444 3049 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.829463 3049 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.832825 3049 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.833470 3049 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.834265 3049 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.836753 3049 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.837349 3049 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.839658 2945 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.840749 2945 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:55.842659 2258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.006s sys 0.008s
W20260927 23:30:55.843185 2258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.006s sys 0.008s
W20260927 23:30:55.844250 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 878753us
I20260927 23:30:55.838110 3049 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.854377 3049 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.853860 2945 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.861850 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:55.864516 3040 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:55.864998 3040 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.865480 3040 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:55.880352 1703 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:55.881292 2899 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.856364 3049 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.882162 2899 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.882802 3051 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:55.887480 2945 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.887775 3051 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:55.887996 3050 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:55.888269 2945 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.888688 3050 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.885632 3034 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.891273 3034 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.891752 2873 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.892527 2873 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.886631 2996 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.896090 2996 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.896534 2893 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:55.897361 2893 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.884857 2865 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:30:55.900751 2291 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51538 (request call id 299) took 1716 ms (client timeout 1966 ms). Trace:
W20260927 23:30:55.901213 2291 rpcz_store.cc:269] 0927 23:30:54.182669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:54.182996 (+ 327us) service_pool.cc:227] Handling call
0927 23:30:55.878125 (+1695129us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b540060fed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:30:55.898813 (+ 20688us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4978560}
I20260927 23:30:55.901441 2865 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.903769 3050 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:55.904366 3050 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.890511 2945 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.906442 2899 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.872713 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 96) took 2440 ms. Trace:
I20260927 23:30:55.901760 2893 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:55.883298 3049 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.907313 2996 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.893950 3034 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:55.904474 2865 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:55.907465 1735 rpcz_store.cc:276] 0927 23:30:53.432199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:53.432430 (+ 231us) service_pool.cc:227] Handling call
0927 23:30:53.718397 (+285967us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:53.751824 (+ 33427us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:55.870946 (+2119122us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b54001dea54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:30:55.872163 (+ 1217us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54001dea50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:30:55.872450 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":695297664}
I20260927 23:30:55.913641 3049 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.888267 3051 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:55.915392 2873 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:55.882288 1703 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:55.918231 3049 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.919157 3049 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.922192 3049 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:55.930626 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 48543.7us
I20260927 23:30:55.935225 2945 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:55.934800 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 48155.3us
I20260927 23:30:55.935979 2945 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.906277 3050 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.945756 3050 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.934681 3049 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:55.938202 2945 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:55.958562 3055 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:55.959457 3055 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.962532 3055 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:55.963816 2981 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:55.964500 2981 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:55.966701 1702 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.553s (SENT) [suppressed 2 similar messages]
I20260927 23:30:55.971820 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:30:55.973057 2956 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.008s user 0.024s sys 0.000s
I20260927 23:30:55.973184 1730 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:55.973538 2956 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.954823 3049 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:55.976048 3049 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:55.984515 2996 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.985664 2996 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:55.984817 2956 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:55.987920 2996 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:55.988044 2891 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:55.988996 2891 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.994536 2945 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.995687 2892 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.995499 2945 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.996407 2892 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.998464 3050 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:55.999246 3050 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:55.999993 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.004259 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.004947 2292 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.006309 2981 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.002027 2892 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:56.002389 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 69741.7us
I20260927 23:30:55.998190 2945 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:56.009900 3050 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:56.012533 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4cc0 after lost signal to thread 399
I20260927 23:30:56.011662 2891 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:56.016510 3049 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.016955 3040 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.029059 3049 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:56.031193 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800239b40 after lost signal to thread 1520
I20260927 23:30:56.032591 2896 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.035017 2896 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:56.036754 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800205160 after lost signal to thread 1612
W20260927 23:30:56.037077 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800205140 after lost signal to thread 1613
I20260927 23:30:56.037355 2896 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:56.038739 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800250800 after lost signal to thread 1687
W20260927 23:30:56.039062 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002507c0 after lost signal to thread 1688
W20260927 23:30:56.040302 1684 debug-util.cc:398] Leaking SignalData structure 0x7b080038aca0 after lost signal to thread 1761
W20260927 23:30:56.040593 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08003961e0 after lost signal to thread 1762
I20260927 23:30:56.041595 2915 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.042333 2915 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:56.047746 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002da380 after lost signal to thread 2318
W20260927 23:30:56.047956 1684 debug-util.cc:398] Leaking SignalData structure 0x7b080031f7e0 after lost signal to thread 2319
W20260927 23:30:56.049535 1684 debug-util.cc:398] Leaking SignalData structure 0x7b08002c1a20 after lost signal to thread 2929
W20260927 23:30:56.049788 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800137dc0 after lost signal to thread 2941
I20260927 23:30:56.050303 2291 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:56.028656 3051 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.051954 1485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 147) took 4007 ms. Trace:
I20260927 23:30:56.053571 3051 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.053146 1482 catalog_manager.cc:5495] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:30:56.054600 1684 debug-util.cc:398] Leaking SignalData structure 0x7b080029e7c0 after lost signal to thread 2979
W20260927 23:30:56.054944 1684 debug-util.cc:398] Leaking SignalData structure 0x7b0800269060 after lost signal to thread 2983
I20260927 23:30:56.055722 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:56.057351 2973 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.057510 1482 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 23:30:56.057996 2973 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.064808 2915 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.065194 2973 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.065366 2915 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.066396 2865 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.066960 3054 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.067081 2865 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.067612 3054 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:56.069770 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 133045us
I20260927 23:30:56.070204 3054 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.071988 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.072818 1588 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.073750 2860 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.074263 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.074501 2860 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.075052 1587 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.076191 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.077229 2860 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.077783 2860 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.078236 3016 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.078620 3011 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.078948 3016 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.079313 3011 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.081696 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.082046 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.082537 2984 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.082702 1660 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.082974 3010 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.083218 2984 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.081802 2983 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.081276 1703 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:56.083895 2948 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.088008 3025 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.088110 2948 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.084159 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.083731 3010 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.086634 2991 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.086951 2971 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.093887 2901 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.087344 2872 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.087468 2976 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.065412 2893 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.062639 2905 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.082700 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.066113 2915 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.091163 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.091260 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.091456 3016 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.091975 2903 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.092115 3026 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:56.089950 1662 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.069018 2865 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.094861 3056 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.101347 2973 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.094648 2901 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.101706 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.101996 3051 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:56.102298 1581 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.102612 3022 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.102782 1703 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:56.104025 2984 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.065398 2888 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.113794 3027 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.113986 2963 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.105084 2983 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.121295 1586 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.095492 2896 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.128206 2875 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.088639 3025 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.127508 1735 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:56.099061 2860 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.103940 2941 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.141482 1657 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.113788 3014 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.094478 2971 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.123342 3051 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.122895 2873 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.128682 2998 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.126564 3054 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.127859 2976 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.148954 2910 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:56.148592 2872 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.115970 2991 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.141702 2865 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.148841 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.149461 3017 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.149571 2905 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.150022 2896 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.213996 2976 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.214536 2901 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.215322 2896 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.150455 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.150709 3015 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.151230 2929 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.151667 2854 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.153133 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.128938 3016 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.127969 1730 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
W20260927 23:30:56.154625 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 51991.2us
I20260927 23:30:56.148851 2938 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.128544 3052 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.202138 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.143486 2984 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.149389 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.165112 3051 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:56.128337 3056 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.213047 2894 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.146653 2888 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.149487 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.213629 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:56.157238 2963 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.212178 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.203715 3040 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.145117 2893 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.157014 2941 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.148284 2973 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.052374 1485 rpcz_store.cc:276] 0927 23:30:52.044227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:52.045909 (+ 1682us) service_pool.cc:227] Handling call
0927 23:30:56.051654 (+4005745us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:30:52.264566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:30:52.270118 (+ 5552us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:30:52.270148 (+ 30us) write_op.cc:435] Acquired schema lock
0927 23:30:52.270165 (+ 17us) tablet.cc:660] Decoding operations
0927 23:30:52.270719 (+ 554us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:30:52.270778 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0927 23:30:52.270810 (+ 32us) tablet.cc:683] Acquiring locks for 3 operations
0927 23:30:52.271155 (+ 345us) tablet.cc:699] Row locks acquired
0927 23:30:52.271178 (+ 23us) write_op.cc:260] PREPARE: finished
0927 23:30:52.271316 (+ 138us) write_op.cc:270] Start()
0927 23:30:52.271415 (+ 99us) write_op.cc:276] Timestamp: P: 1790551852271275 usec, L: 0
0927 23:30:52.271435 (+ 20us) op_driver.cc:331] REPLICATION: starting
0927 23:30:52.341771 (+ 70336us) log.cc:842] Serialized 1331 byte log entry
0927 23:30:53.070072 (+728301us) op_driver.cc:444] REPLICATION: finished
0927 23:30:53.417736 (+347664us) write_op.cc:301] APPLY: starting
0927 23:30:53.417816 (+ 80us) tablet.cc:1365] starting BulkCheckPresence
0927 23:30:53.417960 (+ 144us) tablet.cc:1368] finished BulkCheckPresence
0927 23:30:53.434066 (+ 16106us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:30:53.435605 (+ 1539us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:30:53.435752 (+ 147us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 23:30:53.435792 (+ 40us) write_op.cc:312] APPLY: finished
0927 23:30:53.471606 (+ 35814us) log.cc:842] Serialized 35 byte log entry
0927 23:30:53.471768 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:53.471878 (+ 110us) write_op.cc:454] Released schema lock
0927 23:30:53.471917 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0927 23:30:53.472228 (+ 311us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:53.472257 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":84376,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":347400,"apply.run_cpu_time_us":2933,"apply.run_wall_time_us":54813,"num_ops":3,"prepare.queue_time_us":116287,"prepare.run_cpu_time_us":3218,"prepare.run_wall_time_us":101833,"replication_time_us":798531,"thread_start_us":294819,"threads_started":2}]]}
W20260927 23:30:56.264732 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 51204.1us
I20260927 23:30:56.215772 2915 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.254073 1737 raft_consensus.cc:2354] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:56.255944 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.155900 2903 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.149783 2875 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.217922 3048 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:56.259194 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 48077.2us
I20260927 23:30:56.247570 2860 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.258890 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.259948 2948 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.262992 2906 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.280820 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.256606 3011 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.257910 1735 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:56.262866 3056 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.278074 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.212313 3054 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.283531 1588 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.283556 1703 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:56.279577 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.285756 3062 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.285967 1579 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.211278 3022 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.265182 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.255721 2929 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.266289 1586 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.271382 3067 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:56.271266 2894 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.272178 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.272571 2872 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.249356 2896 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.273347 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.272677 3015 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.274718 1584 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.248800 2873 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.277465 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.276849 1654 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:56.213075 2910 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.278674 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.278964 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.254655 2865 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.208004 3049 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:56.307301 2929 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.307512 2873 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.307682 3049 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.308193 3049 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:56.291962 2983 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.291972 3025 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.275413 2976 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.250067 2998 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.315234 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:56.299779 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 53708.8us
I20260927 23:30:56.265013 3016 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.301043 1661 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:56.301898 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.301278 1585 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.298642 3067 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.273921 2984 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.289340 2901 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:56.283133 3061 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:56.299103 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.304857 1732 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:56.307551 2865 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.266599 2938 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.317559 1588 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.316483 2976 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.277990 3052 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.215922 3027 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.288111 2906 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.327350 2888 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.327741 1583 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.270185 3017 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.275889 2854 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.329532 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.287515 3048 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.332540 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.254724 3014 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.321228 3067 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:56.316768 3064 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.321556 2896 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.328007 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.331698 2893 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.343214 2875 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.336454 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.351631 2894 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.342607 3040 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
W20260927 23:30:56.339612 3065 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260927 23:30:56.352681 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.340358 3039 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:56.344074 2963 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.353349 3039 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:56.344210 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:56.328200 2901 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.354285 1732 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:56.344784 2991 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.346110 1660 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.327764 2973 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.348594 2291 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:56.341073 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.293795 3062 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.340603 2903 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.351044 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.351044 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.353757 3039 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:56.358120 1657 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.328771 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.351661 3064 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.359004 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:56.361071 2290 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:56.361279 3017 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.340241 3015 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.365352 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.368659 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:56.369370 2294 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:56.369401 2896 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.335186 3066 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:56.359648 2998 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.374017 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.375145 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.384284 1657 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.375677 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.375846 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.376701 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.377086 2905 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.377797 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.378041 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.378090 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.379468 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.379717 2901 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.380033 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.380342 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.381242 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.381691 1731 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:30:56.382268 2976 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.382944 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.385157 1661 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.367609 1627 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:56.386812 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:56.389968 1662 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:56.367771 2941 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.386618 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.384868 3054 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.385686 3066 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.348793 3061 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:56.398567 1728 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
W20260927 23:30:56.997650 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 639866us
I20260927 23:30:56.515223 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:56.999053 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.999897 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.441413 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.471261 1735 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:56.403622 1733 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:56.678881 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.680124 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:56.771096 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 80322.8us
W20260927 23:30:57.002276 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 611883us
I20260927 23:30:56.483503 1703 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:30:57.004905 1482 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.013s sys 0.042s
I20260927 23:30:57.001787 1586 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
W20260927 23:30:57.012212 1482 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.726s user 0.014s sys 0.042s
I20260927 23:30:57.008867 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.009099 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.014626 2979 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.014786 3061 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.015034 2979 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:56.386133 3022 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:56.374079 2971 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.016379 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.574870 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.013978 1588 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
W20260927 23:30:56.633388 3070 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20260927 23:30:57.014072 1587 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.000744 1580 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.022344 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.019888 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.022707 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.025449 1663 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.565976 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.026449 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.018914 2979 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.027145 1736 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:57.027344 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.027946 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.028455 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.028704 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.029026 1655 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.029220 1662 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.030025 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.030612 1662 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:30:56.705222 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.470826 2891 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:57.028481 1736 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:30:57.010010 1582 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.031371 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.033685 1662 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.034668 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:57.019371 3049 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.707s user 0.009s sys 0.034s
I20260927 23:30:56.479588 2984 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.552625 3048 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.037380 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:56.678716 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.041481 1627 leader_election.cc:398] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:56.613889 3064 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.043787 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:56.401103 1658 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.546106 3016 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:56.721522 3027 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.040701 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.040940 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.667989 2938 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.057206 1737 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:57.056560 1580 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.042053 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.038909 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.042670 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.040144 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.060065 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.031284 1584 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.043526 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.044423 1627 leader_election.cc:401] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:30:57.011770 3014 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.045006 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.400389 1654 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:56.575137 3031 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:57.046873 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.046149 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:56.655663 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 140143us
I20260927 23:30:57.049275 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.051273 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.070683 1656 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.052276 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.070998 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:56.744084 3067 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:56.655963 3051 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.058993 1588 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.060267 1586 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.080946 1657 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.083849 1656 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
W20260927 23:30:57.078585 1553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.017s sys 0.007s
W20260927 23:30:57.086560 1553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.018s sys 0.007s
I20260927 23:30:57.059543 1733 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:57.061690 1587 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.053326 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.062755 1663 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.031523 2891 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.066313 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.067034 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.069926 1662 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.051954 1583 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
W20260927 23:30:57.105758 1551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.002s sys 0.031s
I20260927 23:30:57.095178 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:57.109969 1551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.906s user 0.002s sys 0.033s
I20260927 23:30:57.108345 1582 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.069871 1659 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.072590 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:57.064889 2290 consensus_meta.cc:287] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.701s user 0.007s sys 0.005s
I20260927 23:30:57.071678 1580 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.073835 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.075279 2910 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.070940 1581 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.071311 1660 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.039976 3049 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.061477 3010 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:57.078365 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57691.5us
I20260927 23:30:57.115562 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:30:57.074649 3039 consensus_meta.cc:287] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.704s user 0.000s sys 0.012s
W20260927 23:30:57.102468 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 188646us
I20260927 23:30:57.117461 1627 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.104879 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.118181 1657 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.088353 3014 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:57.119421 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.119428 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.122254 1660 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:30:57.108875 1730 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:57.122593 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.116917 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.125002 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.124445 1734 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:57.129748 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.112416 2290 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:57.127885 2294 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.133764 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.134173 1627 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.124183 2938 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.080570 3031 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.127583 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.135013 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.127534 2906 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.129463 2935 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:30:57.126284 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.142982 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.143494 1663 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.145661 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.122593 3039 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.151122 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.152144 1657 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:57.134797 3066 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.153271 2906 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.153668 2906 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:57.154031 2906 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.155905 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.156522 1663 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.156514 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.157455 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.157645 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.157990 1663 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.158108 3049 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.158771 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.144984 2935 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:57.160153 2935 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:57.138005 2938 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.160719 2938 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.158847 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.159070 2854 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.162199 1663 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:57.159480 1658 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260927 23:30:57.165549 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 682126us
I20260927 23:30:57.164811 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.167353 1660 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:57.148622 3022 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.165673 1627 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.173200 1703 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.177814 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.179050 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:57.179656 3039 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.180197 3039 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:57.180332 3067 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.180706 3039 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:57.182545 1732 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:57.179663 2290 raft_consensus.cc:1799] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Handling vote request from an unknown peer 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:57.183622 2290 raft_consensus.cc:2379] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:57.183317 2979 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.181639 3039 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.185248 2979 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.185990 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.190223 2258 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.204231 1656 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:57.173539 3022 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:57.187685 3027 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.226799 3027 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.227188 3027 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.210690 3040 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:30:57.227665 3040 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:57.227993 3040 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:57.228153 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.229713 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.230590 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.231205 1657 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.232048 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.232802 1658 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:57.234097 2916 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.166731 2941 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260927 23:30:57.234560 2916 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:57.234947 2916 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.232138 3074 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:57.237633 1663 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:57.209319 1627 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.241645 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.241842 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.242261 1660 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:57.242481 1583 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:57.242738 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.243913 3074 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.245126 2891 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.246021 3074 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:30:57.247871 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 59730.3us
I20260927 23:30:57.249511 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.251332 1662 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
W20260927 23:30:57.253619 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 46006.8us
I20260927 23:30:57.255918 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.207183 3051 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.225726 3022 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.261200 1660 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:30:57.261825 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.260747 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.258394 2941 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.266124 2941 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.233024 3062 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.266517 2941 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.267024 3031 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.270031 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.271030 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.271256 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.271639 1582 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
W20260927 23:30:57.273030 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 123782us
I20260927 23:30:57.276156 2935 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.247391 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.294689 1658 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:57.295485 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.296248 1728 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.297253 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.297725 1655 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:57.300725 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.301195 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.301823 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.302557 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.303027 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.303601 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.304035 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.305590 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.306129 1657 raft_consensus.cc:2362] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4. Re-sending same reply.
I20260927 23:30:57.306913 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.307922 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.308456 1657 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:57.309239 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.305071 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.308110 3027 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.310165 1662 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.310575 1551 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:57.310987 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.311404 3016 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:57.311650 1662 raft_consensus.cc:2354] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3.
W20260927 23:30:57.311556 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 48810.5us
I20260927 23:30:57.312532 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.313122 1663 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.304185 1656 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:30:57.314134 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.314932 3027 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.316784 2938 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.278223 2935 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:57.320498 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.323042 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.323585 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.323746 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.324699 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.324878 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.325426 1582 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:57.326282 3052 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.326165 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.325244 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.309760 2916 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.309765 1657 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.336838 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.335999 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.338236 1580 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:30:57.341033 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.341627 1734 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.337401 1579 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
W20260927 23:30:57.346266 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 151133us
I20260927 23:30:57.326056 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.329367 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.349327 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.350160 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.352698 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.353199 1736 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.357632 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.358228 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.359064 1731 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.360133 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.361496 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.362241 1731 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:57.363438 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.365041 2916 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.365532 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.366532 2941 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.367003 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.368162 3022 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.368487 2941 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.368434 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.370190 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.371273 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.371789 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.372107 1659 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:57.372344 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.373057 1729 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:57.373847 1729 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b90254fab234e29ba132c165c446490"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:57.363538 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.374243 1731 raft_consensus.cc:2396] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:30:57.376768 1729 raft_consensus.cc:595] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:57.377431 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.378228 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.378961 3039 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.379058 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.376530 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.379880 2935 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.380000 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.380249 2935 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:57.380748 2935 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.380882 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:57.382537 2891 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.382902 2891 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:57.383057 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1089) took 1229 ms. Trace:
I20260927 23:30:57.383275 2938 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.383440 1730 rpcz_store.cc:276] 0927 23:30:56.153976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:56.256681 (+102705us) service_pool.cc:227] Handling call
0927 23:30:56.260888 (+ 4207us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:57.126134 (+865246us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:57.382796 (+256662us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b5400230fd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:30:57.382979 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1935596800}
I20260927 23:30:57.350978 1585 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:57.371805 1581 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:57.379371 3039 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:30:57.385113 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 110171us
I20260927 23:30:57.370036 3022 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.382017 1702 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.383280 2891 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.383472 3075 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.407508 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.400774 3039 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.426532 1661 raft_consensus.cc:1261] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:30:57.429641 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.465057 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.464047 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.472677 1660 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:57.470106 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.474047 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.466761 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.479573 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.479568 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:57.450626 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 45085us
W20260927 23:30:57.408205 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 161194us
I20260927 23:30:57.482228 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.475965 1658 raft_consensus.cc:2354] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:57.407119 3074 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.486989 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.493731 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.494905 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.466463 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.493865 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.498040 1659 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:57.450979 3049 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.500388 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.494158 1702 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:57.501060 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.413789 3075 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:57.500334 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.469347 2938 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20260927 23:30:57.492713 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.502498 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:57.505293 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 153535us
I20260927 23:30:57.506753 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.493734 3074 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:57.472244 3040 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.512557 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.501771 3075 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:30:57.689442 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 99372.1us
W20260927 23:30:57.722841 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 228765us
I20260927 23:30:57.499955 3049 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:30:57.726409 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 246760us
I20260927 23:30:57.710924 2935 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.729671 1700 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.732000 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.670354 3022 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:57.725296 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:57.732791 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.721750 3074 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:57.727161 2906 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.571s user 0.009s sys 0.019s
I20260927 23:30:57.735878 1702 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.730365 2935 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:57.716010 3039 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.730901 3049 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:30:57.761283 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 197075us
I20260927 23:30:57.765573 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.764344 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.768090 1734 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
W20260927 23:30:57.727262 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 88007.8us
I20260927 23:30:57.732038 3093 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.760306 2906 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.780606 3093 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:57.781020 3093 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.782194 2906 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.775139 1655 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:30:57.782907 1655 raft_consensus.cc:595] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:57.783254 1655 raft_consensus.cc:603] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:30:57.783680 3039 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:57.765966 3022 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.783938 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.784065 1584 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:57.784509 1736 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.785486 3022 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.776022 3040 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:57.776974 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.786744 1583 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:57.777330 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.772673 2891 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:57.787810 1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 119) took 1428 ms (client timeout 1514 ms). Trace:
I20260927 23:30:57.788161 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:57.788240 1732 rpcz_store.cc:269] 0927 23:30:56.359893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:56.362424 (+ 2531us) service_pool.cc:227] Handling call
0927 23:30:57.787601 (+1425177us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54002bcd54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:30:57.787730 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":398662016}
I20260927 23:30:57.789326 2891 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:57.789944 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.791631 1733 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:57.792060 3054 raft_consensus.cc:3032] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:57.792572 3054 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:57.795070 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:30:57.779557 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 154092us
I20260927 23:30:57.795742 1663 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:30:57.797147 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.807574 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:57.818226 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 48052.4us
I20260927 23:30:57.819661 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.819730 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.822815 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.824911 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.825181 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.822381 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.826485 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.828420 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.818429 1654 raft_consensus.cc:2354] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:57.835677 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.793639 3054 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.820621 3092 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:30:57.840008 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 321580us
I20260927 23:30:57.845081 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.846683 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:57.847093 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 111343us
I20260927 23:30:57.828325 2935 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.848611 1580 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:57.847470 1587 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:57.849016 1663 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:57.837754 3039 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.851606 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:57.853235 1661 raft_consensus.cc:2354] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
W20260927 23:30:57.853569 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 57709.5us
I20260927 23:30:57.855206 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.855818 1581 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.847867 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.856169 3049 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.839797 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.837754 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.852403 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.819317 3085 raft_consensus.cc:979] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:57.852718 1702 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.846801 3092 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:57.863641 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.864614 2257 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:57.851806 3054 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:57.846292 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 57849.9us
I20260927 23:30:57.867877 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.858959 2935 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.858186 3049 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:57.852114 3039 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.887259 3039 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.887106 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.887952 1587 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.876345 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.882908 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:57.866293 3092 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.886473 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:57.896034 1733 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:57.869690 3099 raft_consensus.cc:1067] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:57.882735 3085 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.885632 2935 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.887276 3040 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.902664 1551 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:57.868696 3091 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260927 23:30:57.904690 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.895071 2906 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:57.908157 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.927488 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.929605 1582 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.917831 1587 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:57.931663 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.918035 3092 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:57.918399 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:30:57.919258 2906 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:57.921610 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.922364 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.922652 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.903517 3085 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:57.905655 3040 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:57.958173 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.923053 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.909304 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.934914 1581 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:57.927340 2910 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:57.933571 1628 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:57.960212 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.906859 3103 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:57.976320 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:30:57.974606 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 244984us
I20260927 23:30:57.977274 1551 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:57.982863 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:57.978367 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.981211 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.983965 1700 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:30:57.977512 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 112952us
W20260927 23:30:57.985309 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 81195.6us
I20260927 23:30:57.985466 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:30:57.985462 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51994.5us
I20260927 23:30:57.986443 1702 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.987349 1700 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:57.989128 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:57.991432 1729 raft_consensus.cc:2354] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:57.990840 3054 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:57.993197 3102 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.991695 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:57.993989 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:57.958371 3085 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.960806 3040 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.999507 1702 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:57.995512 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:57.959296 1584 raft_consensus.cc:595] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:57.975390 3103 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.012637 1628 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:57.980592 3016 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:57.975912 1585 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:58.019930 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.020105 1628 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.022012 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:58.003224 3099 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.022877 3099 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.023305 3099 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.023828 3054 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.024225 3054 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:58.024571 3054 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.026400 3109 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.019583 3016 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:58.027192 3016 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:57.999289 2905 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:57.980579 3015 consensus_queue.cc:1035] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 23:30:57.993587 3102 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:58.042518 3102 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.027910 1551 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:58.047590 1551 leader_election.cc:398] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:58.047874 1551 leader_election.cc:401] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:30:58.048048 2906 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.049046 3066 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.049439 3066 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:58.049762 3066 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.050235 3092 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.012900 2891 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.050904 2891 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.051254 2891 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.033002 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.033690 1582 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:58.033684 3054 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.018194 3103 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.037384 3040 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.006577 3051 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.076275 1552 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.082743 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.083021 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.020476 2901 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.083837 2289 raft_consensus.cc:2354] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:58.083717 2935 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.085532 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.086244 2287 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:58.086819 3061 raft_consensus.cc:2741] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:30:58.087461 1728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:30:58.088040 1552 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.088074 1728 raft_consensus.cc:595] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.077862 2906 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:58.000313 3108 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.085650 2935 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.021556 2938 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.083480 2290 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:58.018700 3093 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.109992 1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48976 (request call id 244) took 5878 ms. Trace:
I20260927 23:30:58.115758 1700 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:58.117813 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.118151 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.117405 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.161667 2293 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:58.110646 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 23:30:58.117012 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.130774 3108 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.028043 3052 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:30:58.116209 2901 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:30:58.209476 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 93793.9us
I20260927 23:30:58.163666 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:30:58.090458 3031 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:30:58.081864 3051 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:58.159247 1485 catalog_manager.cc:5495] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 3, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:30:58.134743 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 72170.3us
I20260927 23:30:58.213922 2935 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.124811 3054 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.219025 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:58.112126 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.087441 3072 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.214510 1700 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.158601 2938 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:58.115778 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.115381 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.118633 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:30:58.221556 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 133576us
I20260927 23:30:58.217240 1485 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:58.223070 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.231230 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.225059 1700 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:58.218628 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.232255 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.156126 2905 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.114259 3016 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.113525 3066 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.213065 3108 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.109004 2891 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.228814 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.222607 2935 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.232180 2293 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:58.081283 3015 consensus_queue.cc:1046] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.149s
I20260927 23:30:58.234946 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.235704 1657 raft_consensus.cc:1226] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.242169 2257 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.160951 1482 rpcz_store.cc:276] 0927 23:30:52.231164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:52.231331 (+ 167us) service_pool.cc:227] Handling call
0927 23:30:58.109900 (+5878569us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:30:57.103789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:30:57.233234 (+129445us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:30:57.233286 (+ 52us) write_op.cc:435] Acquired schema lock
0927 23:30:57.233303 (+ 17us) tablet.cc:660] Decoding operations
0927 23:30:57.233900 (+ 597us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:30:57.233992 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0927 23:30:57.234033 (+ 41us) tablet.cc:683] Acquiring locks for 3 operations
0927 23:30:57.237626 (+ 3593us) tablet.cc:699] Row locks acquired
0927 23:30:57.251364 (+ 13738us) write_op.cc:260] PREPARE: finished
0927 23:30:57.278058 (+ 26694us) write_op.cc:270] Start()
0927 23:30:57.278217 (+ 159us) write_op.cc:276] Timestamp: P: 1790551857277995 usec, L: 0
0927 23:30:57.278252 (+ 35us) op_driver.cc:331] REPLICATION: starting
0927 23:30:57.299241 (+ 20989us) log.cc:842] Serialized 1280 byte log entry
0927 23:30:57.718927 (+419686us) op_driver.cc:444] REPLICATION: finished
0927 23:30:57.979993 (+261066us) write_op.cc:301] APPLY: starting
0927 23:30:57.994338 (+ 14345us) tablet.cc:1365] starting BulkCheckPresence
0927 23:30:58.011470 (+ 17132us) tablet.cc:1368] finished BulkCheckPresence
0927 23:30:58.011648 (+ 178us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:30:58.012259 (+ 611us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:30:58.012488 (+ 229us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 23:30:58.012546 (+ 58us) write_op.cc:312] APPLY: finished
0927 23:30:58.047147 (+ 34601us) log.cc:842] Serialized 35 byte log entry
0927 23:30:58.047339 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:58.047426 (+ 87us) write_op.cc:454] Released schema lock
0927 23:30:58.047496 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0927 23:30:58.047788 (+ 292us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:58.047831 (+ 43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":51933,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":211719,"apply.run_cpu_time_us":3284,"apply.run_wall_time_us":102447,"num_ops":3,"prepare.queue_time_us":778456,"prepare.run_cpu_time_us":9904,"prepare.run_wall_time_us":288191,"raft.queue_time_us":249910,"raft.run_cpu_time_us":37593,"raft.run_wall_time_us":99715,"replication_time_us":440568,"thread_start_us":1136548,"threads_started":4,"wal-append.queue_time_us":90035}]]}
I20260927 23:30:58.236065 2292 raft_consensus.cc:2379] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:58.232847 1485 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 2 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:30:58.119048 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.123541 3074 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.231596 2285 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:58.246338 3110 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.248600 3040 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.248870 3110 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.249296 3074 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:30:58.250137 3075 consensus_meta.cc:287] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.523s user 0.006s sys 0.003s
I20260927 23:30:58.250545 3075 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.251348 3098 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.241976 2935 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.251986 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.252458 3075 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.239336 3031 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:58.117487 2259 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.253769 3031 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.254603 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:30:58.254977 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 137562us
I20260927 23:30:58.234099 3104 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.256017 3104 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.256388 3104 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.256578 2259 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:58.234817 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:58.237615 2901 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.259532 3093 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.260267 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.259734 3085 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.260757 3084 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.262595 1485 catalog_manager.cc:5495] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: config changed from index 2 to 4, 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:30:58.219990 3051 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.231715 2938 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.241575 1700 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:30:58.303623 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 68748.7us
I20260927 23:30:58.304422 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.237947 3010 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.305498 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:30:58.306429 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 59541.7us
I20260927 23:30:58.261696 3040 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.226045 3099 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.315331 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.309818 3072 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.322986 3072 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.323180 2286 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:58.308691 3073 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.238983 2288 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:58.309604 3061 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 23:30:58.318770 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 74653.2us
I20260927 23:30:58.310657 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.334249 2292 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:58.334431 2257 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.307909 3010 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.335616 3061 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.336061 3061 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.249328 3110 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:30:58.319629 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 78243.4us
I20260927 23:30:58.247025 3117 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:58.323441 3072 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.338982 1700 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.310377 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.346547 2257 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.346920 1700 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.348246 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.317402 3040 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.303236 3039 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.352313 1552 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.353308 2286 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.353502 2891 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.323984 2901 consensus_queue.cc:1046] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260927 23:30:58.354063 2292 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:30:58.356750 3092 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.351302 1700 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.361605 3016 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.381021 3114 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.381284 2901 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.381419 3114 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.381809 3114 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.381907 2901 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.334121 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:58.358552 3075 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.362838 2854 raft_consensus.cc:979] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:58.366082 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.386891 1700 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.387341 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.388195 1700 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:58.388752 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.377676 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.380023 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:58.374766 3092 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.338977 3010 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.390837 1552 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:30:58.382627 3116 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260927 23:30:58.359596 3098 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.391698 2891 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.363611 3040 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.403239 1552 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.404516 1552 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.382586 2901 raft_consensus.cc:3032] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
W20260927 23:30:58.400588 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 72702.4us
I20260927 23:30:58.363624 3117 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:58.392761 3074 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.385987 3075 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.414595 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:58.424712 3075 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.398666 3092 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.418993 3123 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.424152 1700 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.396170 2906 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.394852 3016 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.407135 3015 raft_consensus.cc:1067] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:30:58.400646 2854 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.391547 2938 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.426605 2257 leader_election.cc:398] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3.
I20260927 23:30:58.441244 2257 leader_election.cc:401] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:30:58.425745 1654 raft_consensus.cc:2354] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:58.374707 2935 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.426646 3074 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.363493 3103 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.433795 3073 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.431665 3123 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.403254 2891 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.453357 1700 leader_election.cc:398] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:58.455041 1551 leader_election.cc:398] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:58.433336 3066 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.420212 3097 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.405709 2901 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:58.437435 3120 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:58.436017 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.440239 3049 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.456166 3096 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.469130 3096 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.469062 3120 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.457674 3074 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.463023 2891 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.471282 3120 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.469627 3096 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.471689 3049 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.472096 3049 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.474694 1658 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.474987 3117 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.475667 2938 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:30:58.466657 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 60242.6us
I20260927 23:30:58.442200 3016 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:30:58.513631 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 58740.1us
I20260927 23:30:58.515260 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.443308 3093 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.511130 2257 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.482813 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.496692 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.517778 1729 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.490209 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:58.520726 1660 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:58.520748 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.520774 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.521749 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:58.522042 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.522552 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.522796 3103 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.524428 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.524762 1661 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:58.515444 1657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:30:58.525722 1657 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:58.526146 1657 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.527144 1659 raft_consensus.cc:2354] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
I20260927 23:30:58.528188 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.529539 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.530018 1663 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.531186 1735 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.532130 1729 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.527254 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.533737 1731 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.533826 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.463660 3123 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.509369 3085 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.464088 3031 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3.
I20260927 23:30:58.535367 1659 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:58.535594 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.541358 1660 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.463908 3084 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.538069 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.539729 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.542387 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.536727 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.543464 1663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b90254fab234e29ba132c165c446490"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46200
W20260927 23:30:58.543524 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 139041us
I20260927 23:30:58.546931 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.543982 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.477872 3097 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.548769 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.549711 1733 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.552212 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.573342 1734 raft_consensus.cc:1226] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.552647 3091 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.552799 1735 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.553035 1629 leader_election.cc:398] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:58.575048 1629 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.533885 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.579272 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.531774 3099 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.599658 1629 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.569909 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.574844 2257 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.601009 1736 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:58.539278 3016 consensus_queue.cc:1046] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260927 23:30:58.577576 1657 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.568131 3093 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.600256 1661 raft_consensus.cc:2379] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:58.569278 1658 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:30:58.602694 1657 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.603768 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.537743 3129 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.611937 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.613358 1552 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.614846 1629 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.616312 3128 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.616817 1737 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260927 23:30:58.616931 3128 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.617347 3128 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.617658 3128 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.617856 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.619793 1662 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.610220 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.618829 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:58.621994 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 47223.8us
I20260927 23:30:58.623121 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.621819 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.584604 1656 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:58.625452 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.622359 1629 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.626042 1734 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.553803 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:58.579552 3031 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.554069 3039 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.580236 3091 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.573786 3097 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.605902 3073 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.631675 3022 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.634516 2257 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.632175 1629 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.609790 2938 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:30:58.623744 3102 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.581s user 0.001s sys 0.009s
W20260927 23:30:58.633395 1702 leader_election.cc:334] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.544s (SENT)
I20260927 23:30:58.610352 3093 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.619289 3014 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.618884 3054 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.610970 3129 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.621755 3084 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.647394 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.616819 3066 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:58.649571 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.629431 3126 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.652604 1552 leader_election.cc:398] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3.
I20260927 23:30:58.624957 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:58.656512 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.623030 3104 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.634919 3091 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.629364 3099 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:58.609367 3061 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.659562 2935 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.666437 1629 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.646677 2938 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.650835 3022 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.662276 3129 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:58.678148 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 44810.5us
I20260927 23:30:58.656278 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.635272 3074 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.652632 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.662897 3066 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:30:58.663574 3084 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.679862 1629 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:58.706596 1629 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:30:58.680378 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 57595.7us
I20260927 23:30:58.651895 3016 consensus_queue.cc:1046] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260927 23:30:58.708377 3016 raft_consensus.cc:3032] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:58.708858 3016 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:58.651714 3031 consensus_queue.cc:1046] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20260927 23:30:58.709640 3091 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.649320 3102 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.712103 3091 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.713079 3102 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.714022 1729 raft_consensus.cc:2354] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:30:58.628104 2901 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.715159 2901 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:58.716248 3125 consensus_queue.cc:1159] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a3317e8da742d9aefe4c251abd4566" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
W20260927 23:30:58.701690 1702 leader_election.cc:334] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.617s (SENT)
I20260927 23:30:58.717020 3123 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.709939 3016 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.720376 3016 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.721552 3123 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.723474 1658 raft_consensus.cc:2379] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:58.723613 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.676301 3099 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.724220 1736 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:58.725112 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.683269 3103 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.726246 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.676141 3114 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.727293 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.727560 3104 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.728341 3114 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.729027 3114 consensus_queue.cc:1046] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:30:58.729096 3051 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.704717 3066 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.730252 1737 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:30:58.731132 3130 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260927 23:30:58.734947 1728 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.735973 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.736292 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.737589 2905 raft_consensus.cc:3032] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:58.738029 3084 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:58.740767 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.726049 3099 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:58.747438 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.748611 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.748908 2294 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.737497 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.742250 1629 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:58.744801 1734 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:30:58.750294 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 77269.2us
I20260927 23:30:58.742972 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.738041 2905 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:58.751911 1657 raft_consensus.cc:2354] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:58.682462 2938 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.753158 1736 raft_consensus.cc:2379] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:58.756019 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.689244 3120 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.758136 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:58.759681 2294 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.760617 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.691941 3134 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.757293 3054 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.795807 3104 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.796082 3054 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.796275 3104 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:58.796741 3054 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.796972 3120 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:58.796994 3104 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.797492 3120 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:58.705205 3075 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.756785 3072 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.796602 1657 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:30:58.793752 1702 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.632s (SENT)
I20260927 23:30:58.796281 1700 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.802958 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.804379 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:58.783542 1656 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:58.807045 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56003.4us
W20260927 23:30:58.808128 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 143373us
I20260927 23:30:58.735005 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:58.817413 1700 leader_election.cc:398] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
I20260927 23:30:58.818186 1700 leader_election.cc:401] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260927 23:30:58.815634 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:58.749598 3022 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.759327 3014 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:30:58.827147 1702 leader_election.cc:334] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.657s (SENT)
I20260927 23:30:58.832859 1702 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.807466 3103 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:30:58.841761 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:58.842898 1628 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:58.810938 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.815929 1732 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
W20260927 23:30:58.924490 3108 consensus_meta.cc:287] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.658s user 0.005s sys 0.005s
I20260927 23:30:58.979276 1581 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:58.832101 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:58.830165 3135 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:30:58.947315 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 120199us
I20260927 23:30:58.957216 1588 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:58.738089 3096 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.968328 1657 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:30:58.825088 3139 raft_consensus.cc:3032] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:58.816484 3102 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.922703 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:30:58.950783 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 189440us
W20260927 23:30:58.934703 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 93002.7us
I20260927 23:30:59.000451 1700 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.839882 1730 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:58.849298 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 165741us
W20260927 23:30:58.923066 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 75582.8us
I20260927 23:30:58.933948 3117 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:58.973963 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:58.826689 2935 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:58.948690 3131 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:58.851603 3039 consensus_queue.cc:1046] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.120s
I20260927 23:30:58.983969 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.027169 1588 raft_consensus.cc:3032] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
W20260927 23:30:58.982889 1702 leader_election.cc:334] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.676s (SENT)
I20260927 23:30:58.932740 3092 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:58.969008 3014 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:59.004117 1700 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:58.966576 3022 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:58.933614 3118 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260927 23:30:58.973246 3051 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:58.973176 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:59.025403 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 283165us
I20260927 23:30:59.048393 1629 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:59.026739 2293 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.027468 1588 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.029953 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:59.050617 1586 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:30:59.001673 3061 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.984257 3072 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:58.993618 3139 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:59.025365 3085 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.056937 3157 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:30:59.056972 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:30:59.062438 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 300107us
W20260927 23:30:59.058640 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 54694.9us
I20260927 23:30:59.032020 3148 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:58.981992 3146 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:30:59.033862 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51025.4us
I20260927 23:30:58.986773 3108 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.036974 3014 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:30:59.069106 1702 leader_election.cc:334] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.728s (SENT)
I20260927 23:30:59.026556 2935 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:59.062654 3075 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.074719 2935 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.074922 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.075251 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:58.969108 3137 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
I20260927 23:30:59.061126 3022 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:59.050787 1588 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:58.979147 2905 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.049011 3131 raft_consensus.cc:686] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:59.076033 1730 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.061057 2289 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:59.080466 3049 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.608s user 0.010s sys 0.003s
I20260927 23:30:59.060029 1583 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.078258 3014 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.066228 1700 leader_election.cc:398] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:59.052772 3066 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:59.085179 3141 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.122s user 0.000s sys 0.000s
I20260927 23:30:59.085181 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:59.110764 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:30:59.086709 2891 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.616s user 0.010s sys 0.000s
W20260927 23:30:59.113051 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 66095.2us
I20260927 23:30:59.069862 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.113824 1586 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:30:59.087410 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75443.5us
I20260927 23:30:59.061915 3051 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.092694 1735 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.099290 1700 leader_election.cc:401] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:30:59.087474 1588 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.114377 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:59.116791 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.090468 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:30:59.118844 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 61978.1us
I20260927 23:30:59.119966 1587 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
W20260927 23:30:59.122712 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 56664.8us
I20260927 23:30:59.121068 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.103404 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.138144 1730 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.137979 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.116151 3102 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.139243 1655 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.140553 1660 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:30:59.124444 2905 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.145861 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:30:59.146488 1586 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:59.147544 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:30:59.071437 3157 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.119557 3075 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.098403 3152 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:59.126259 3103 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.108629 3120 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.161387 3120 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.125963 3093 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:59.162536 3010 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.759s user 0.013s sys 0.000s
I20260927 23:30:59.162449 2935 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.163021 3010 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.139111 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.135936 3051 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:59.113878 3139 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.153900 1732 raft_consensus.cc:2379] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.141892 3099 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.159461 3066 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:59.155133 3096 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.103531 3049 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.164762 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.118603 2891 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.163826 3093 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.081980 3146 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.156082 3152 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:30:59.148088 3153 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260927 23:30:59.173223 3139 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.169163 3120 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260927 23:30:59.179386 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.176973 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:59.141103 3084 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.191507 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.186546 1582 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.187701 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.192436 1586 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:59.193210 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.194227 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.194883 1586 raft_consensus.cc:2354] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:59.193219 3049 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.172915 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.196588 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.197427 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.170935 3158 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.200248 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.171623 3092 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.202924 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.206005 3074 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.206676 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.169287 3102 consensus_queue.cc:1046] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260927 23:30:59.208686 3084 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.208985 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:59.209242 2292 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.209897 3140 raft_consensus.cc:3032] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.211550 3157 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.215114 1582 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:59.223964 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:59.191121 2289 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.184935 3152 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:30:59.199415 3040 consensus_meta.cc:287] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.780s user 0.010s sys 0.000s
I20260927 23:30:59.219937 3148 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:30:59.235977 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 121694us
I20260927 23:30:59.237690 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.245260 1628 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:30:59.227017 3129 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.524s user 0.007s sys 0.003s
I20260927 23:30:59.246171 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.198930 2935 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.238874 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:59.243892 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.251708 1581 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:59.240767 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.247529 1554 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.221135 3108 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.254491 1579 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:59.237486 1732 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.257962 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.258810 2891 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.259436 1733 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.259572 1732 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.259815 3010 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:59.260588 1733 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.260658 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51538 (request call id 321) took 1146 ms. Trace:
I20260927 23:30:59.260800 1732 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.261112 2291 rpcz_store.cc:276] 0927 23:30:58.113840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.116788 (+ 2948us) service_pool.cc:227] Handling call
0927 23:30:59.260483 (+1143695us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b540060fed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:30:59.260612 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":626201856}
I20260927 23:30:59.261762 3146 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:59.262004 3106 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:30:59.262845 2285 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.263039 3075 raft_consensus.cc:3032] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.263077 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.263964 1579 raft_consensus.cc:2379] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:30:59.263542 3075 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:59.264664 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.264850 3075 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.264995 3148 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.265733 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.266826 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.267819 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.167402 3131 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.268824 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.236454 3140 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:59.261780 1733 raft_consensus.cc:1226] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.270237 1628 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:59.201025 3054 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.239310 1728 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.243568 3052 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:30:59.275610 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.277969 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.270440 1582 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:59.244115 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.277316 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.251314 3158 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:30:59.286540 3015 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:30:59.287415 3010 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:59.277261 3106 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:30:59.289858 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.290621 2906 consensus_queue.cc:1046] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:59.291538 3126 consensus_queue.cc:1046] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:59.309384 3164 raft_consensus.cc:2741] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260927 23:30:59.309684 3126 raft_consensus.cc:3032] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.310159 3126 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.310470 3073 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.310540 3137 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:59.310961 3126 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.312206 3126 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.266132 3075 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.265846 1585 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:59.276939 3097 consensus_meta.cc:287] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.619s user 0.010s sys 0.000s
I20260927 23:30:59.320168 3014 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.232785 3104 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.297607 3158 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.299624 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.301874 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.308004 3106 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.303428 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.340152 1582 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.273789 3022 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.311866 3137 consensus_queue.cc:1046] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:30:59.293046 3066 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:30:59.315801 2906 raft_consensus.cc:3032] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.344863 3066 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.345463 3066 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.320669 3097 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.346333 3066 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.347506 3103 raft_consensus.cc:979] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: : Instructing follower 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:59.344982 2906 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.348045 1735 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.285280 1733 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.276800 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:30:59.348747 2906 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.349459 1733 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.349532 1731 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.349707 1586 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:59.304996 3109 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260927 23:30:59.344166 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.347863 3097 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.288340 3146 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:30:59.351104 1733 raft_consensus.cc:1226] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.339023 1581 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.257659 3072 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.338352 1588 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:59.339922 1663 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:30:59.355288 1729 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.276110 2938 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.305154 1583 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:30:59.342310 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.298620 2901 consensus_queue.cc:1046] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260927 23:30:59.347464 3066 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.349817 2906 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.352895 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.251344 3040 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.354204 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.354622 3146 raft_consensus.cc:3032] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.274497 3140 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.361563 3075 raft_consensus.cc:1067] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Signalling peer 0fe98f486bab4d0b9c421c145d83f1ce to start an election
I20260927 23:30:59.352579 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:30:59.339519 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 69352.8us
W20260927 23:30:59.353404 3128 consensus_meta.cc:287] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.735s user 0.012s sys 0.000s
I20260927 23:30:59.360810 3137 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.366997 1585 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.341926 3164 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:30:59.368089 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:59.372596 1733 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.351099 3109 raft_consensus.cc:3032] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.355670 1663 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:30:59.255800 3129 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:59.378012 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 314854us
I20260927 23:30:59.357651 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:30:59.388482 2938 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.373319 3108 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.409927 2292 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:30:59.409885 1627 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.409436 1734 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.394629 1702 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.378367 3146 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.371225 3015 consensus_queue.cc:1046] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
W20260927 23:30:59.427999 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 308063us
I20260927 23:30:59.390969 3137 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.393824 3164 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:30:59.431703 1551 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.409250 3131 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.422745 3109 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:30:59.428440 1730 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.432258 2938 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260927 23:30:59.444882 3014 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.445495 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.445899 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.446219 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.446718 2290 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:59.446957 2293 raft_consensus.cc:2379] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:59.440129 2286 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.447824 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:30:59.448422 2285 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:30:59.433089 3161 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.449462 3161 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.442226 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.442649 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.433152 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.433779 1702 leader_election.cc:398] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3.
I20260927 23:30:59.454102 2292 raft_consensus.cc:2379] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:59.445616 3040 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:30:59.456437 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.450629 2288 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.450289 3161 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.440065 3167 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260927 23:30:59.424098 3158 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.459358 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.459592 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.469043 1736 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:30:59.469251 1733 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.459959 3125 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.435853 3061 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
W20260927 23:30:59.442945 3110 consensus_meta.cc:287] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.075s user 0.009s sys 0.009s
I20260927 23:30:59.433135 3022 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.436414 3072 raft_consensus.cc:3032] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.476992 3072 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.450713 3016 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:59.464676 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.431788 3054 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.439245 3164 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.426640 3162 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 23:30:59.469849 1627 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.473819 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:30:59.476404 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.429849 3010 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260927 23:30:59.467816 3074 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.464490 1628 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:59.433712 3108 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.434914 3104 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.483206 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.394512 3128 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.484403 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.462044 3129 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.487315 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.488152 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.498816 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.523193 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.524184 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.498513 3072 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.499890 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.502246 3054 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:30:59.391127 3142 consensus_queue.cc:1159] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a3317e8da742d9aefe4c251abd4566" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 23:30:59.654516 1580 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.510954 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.512382 2291 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:30:59.473311 3061 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.471354 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:30:59.678221 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.514794 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.679201 1657 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 23:30:59.679420 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 164733us
I20260927 23:30:59.514459 3161 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.517454 3174 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.474795 3110 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:59.511142 3135 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.000s
I20260927 23:30:59.511579 3075 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:30:59.655730 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 144850us
I20260927 23:30:59.510309 3016 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.676738 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.470696 3014 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260927 23:30:59.476398 3166 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.229s
W20260927 23:30:59.681293 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 216860us
I20260927 23:30:59.687708 1657 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.688334 1702 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:59.675576 3143 consensus_queue.cc:1046] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260927 23:30:59.480896 3125 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260927 23:30:59.657724 3158 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:30:59.482854 3146 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.522066 3162 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:30:59.694585 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.695752 1654 raft_consensus.cc:2379] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:30:59.683903 3117 raft_consensus.cc:3032] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.676455 3074 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.702438 2257 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.704676 1629 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:59.707621 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.689211 3072 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.683326 3073 raft_consensus.cc:3032] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:30:59.523866 3052 raft_consensus.cc:686] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.706547 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:30:59.482707 3137 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.718111 2257 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.715312 3117 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:30:59.715651 2287 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.720057 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:30:59.712764 3171 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.721032 1654 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.721565 1657 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
W20260927 23:30:59.721966 3169 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.259s user 0.000s sys 0.001s
I20260927 23:30:59.689677 3010 consensus_queue.cc:1046] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260927 23:30:59.689394 3135 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
W20260927 23:30:59.739223 3168 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.356s user 0.000s sys 0.001s
I20260927 23:30:59.740046 3110 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:30:59.744705 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.686946 3167 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260927 23:30:59.518267 3109 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.763983 1551 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.765249 3176 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260927 23:30:59.709848 3075 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 23:30:59.767807 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 49775us
I20260927 23:30:59.714179 1581 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:30:59.715600 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:30:59.711340 3162 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:30:59.711606 3152 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.776968 1735 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:30:59.744081 2938 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.718114 3181 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:59.734687 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:30:59.796726 1702 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.817236 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:30:59.799239 2257 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:30:59.799734 3132 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.095s user 0.000s sys 0.014s
I20260927 23:30:59.772351 3146 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.770787 3108 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:30:59.794013 3104 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:30:59.808079 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 103484us
W20260927 23:30:59.809221 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 45319.8us
I20260927 23:30:59.765482 3052 consensus_queue.cc:237] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.775594 3171 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:30:59.810166 1663 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.829946 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.798065 3110 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.744726 3073 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:30:59.806538 3135 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260927 23:30:59.789031 3106 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.807667 3016 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.835551 1586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:46522
I20260927 23:30:59.956445 1730 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.046597 3192 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:30:59.960168 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 78947.5us
I20260927 23:31:00.055979 1551 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.055683 3192 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.878072 3129 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:30:59.919242 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 213469us
I20260927 23:30:59.923530 3128 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.988065 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.033963 1702 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:30:59.854995 3163 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:59.833333 1732 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:00.035645 1662 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 23:30:59.854795 3066 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.875761 2291 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:00.057915 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.833596 3117 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.027623 3165 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:00.155546 3165 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.158638 3066 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.159382 3066 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:00.159956 3066 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.162901 3074 consensus_queue.cc:1046] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:30:59.915568 3189 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:30:59.863184 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 183154us
I20260927 23:30:59.919013 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:00.167220 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:00.167526 1737 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:00.167815 3066 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:30:59.924764 3182 raft_consensus.cc:3032] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.168578 3182 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.169636 3182 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.170910 3140 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.171303 3182 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.061661 1586 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:00.173108 1586 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.174206 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1380) took 1327 ms. Trace:
I20260927 23:31:00.174612 1728 rpcz_store.cc:276] 0927 23:30:58.846898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.849163 (+ 2265us) service_pool.cc:227] Handling call
0927 23:30:59.217944 (+368781us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.297012 (+ 79068us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:00.174018 (+877006us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54003c3050. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:00.174150 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":535963520}
I20260927 23:31:00.175942 3143 raft_consensus.cc:3032] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.176288 3143 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.177006 3143 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.857044 3010 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.206s
I20260927 23:30:59.891341 3162 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:30:59.930990 3173 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.486s user 0.002s sys 0.003s
I20260927 23:31:00.057869 3192 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.028249 3164 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.183602 1586 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:00.092092 3196 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260927 23:31:00.066754 3073 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:30:59.916869 3015 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.151s
I20260927 23:31:00.191644 3108 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:30:59.885885 3184 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:30:59.889642 2910 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.191169 3163 raft_consensus.cc:686] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:00.198058 2910 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.198930 2910 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.198853 3163 consensus_queue.cc:237] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.919416 1735 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.859052 3075 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.071658 3106 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.074240 3148 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.201025 3148 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.202071 3148 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:30:59.852084 3181 raft_consensus.cc:686] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.204150 3181 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:30:59.862453 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 156268us
I20260927 23:31:00.209492 2259 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:00.211844 2906 consensus_meta.cc:287] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.833s user 0.012s sys 0.000s
W20260927 23:31:00.216804 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 47885.9us
I20260927 23:31:00.054111 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.220403 1586 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.062143 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1121) took 1115 ms. Trace:
I20260927 23:31:00.078719 2292 raft_consensus.cc:2379] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:30:59.832690 3031 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.083832 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:30:59.962139 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1382) took 1028 ms. Trace:
I20260927 23:30:59.990304 1657 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.920084 3155 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:31:00.117048 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 287187us
I20260927 23:31:00.152195 1731 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:00.152415 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.230840 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.154490 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.172515 3145 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 23:31:00.179633 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 145775us
W20260927 23:31:00.180553 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 122713us
I20260927 23:31:00.238942 1700 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:00.240041 1700 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:30:59.920229 1658 raft_consensus.cc:1226] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:30:59.857362 3171 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:30:59.901278 2905 raft_consensus.cc:3032] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.051019 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.191934 3102 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:31:00.196907 3073 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:30:59.926370 3152 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:00.224510 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 168845us
I20260927 23:30:59.901124 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.221691 1734 rpcz_store.cc:276] 0927 23:30:58.934158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.934282 (+ 124us) service_pool.cc:227] Handling call
0927 23:30:59.409231 (+474949us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.423717 (+ 14486us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:59.961910 (+538193us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54001d1d50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:30:59.962070 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1039191424}
W20260927 23:31:00.226817 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 170923us
W20260927 23:31:00.227164 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 60021.5us
I20260927 23:31:00.158164 3165 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.229305 3142 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.221100 1587 rpcz_store.cc:276] 0927 23:30:58.947171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.948527 (+ 1356us) service_pool.cc:227] Handling call
0927 23:30:58.948815 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.120382 (+171567us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:00.061837 (+941455us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400465d54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:00.062044 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":393223552}
I20260927 23:31:00.230176 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.277009 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.231197 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.231765 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.279305 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.280566 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.235100 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.192010 3167 raft_consensus.cc:3032] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.149423 3183 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.067163 3153 raft_consensus.cc:3032] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.242774 3075 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:00.232904 3184 raft_consensus.cc:686] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.234721 3189 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.010s
I20260927 23:31:00.249929 1663 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.249660 3108 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:00.233556 3120 raft_consensus.cc:2741] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260927 23:31:00.308332 1734 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.235625 3096 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 23:31:00.246209 3072 consensus_meta.cc:287] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.500s user 0.008s sys 0.000s
I20260927 23:31:00.268523 1627 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.268642 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.309410 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:00.275961 3125 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:00.270661 1585 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.315158 1737 raft_consensus.cc:3032] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.270725 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.271090 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.272253 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.255602 2905 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.273952 3198 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:00.186616 3128 consensus_queue.cc:1046] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260927 23:31:00.285074 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.299130 3153 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.286267 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.323383 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.324357 1659 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:00.287859 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.289938 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:00.325345 1586 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.291594 3165 consensus_queue.cc:1046] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260927 23:31:00.252998 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 43652us
I20260927 23:31:00.294803 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:00.294870 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.295302 3097 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:00.254452 3171 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:00.267989 3131 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.317299 3152 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.307349 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.369740 2291 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.284163 3145 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.277132 3146 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.283960 3167 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:00.315176 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 46719.5us
I20260927 23:31:00.376657 1627 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.307565 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.377737 1735 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:00.378260 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 103377us
I20260927 23:31:00.315487 1737 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:00.379401 1737 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.316524 3174 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260927 23:31:00.316931 1700 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.303153 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.329315 1659 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 3. Ops: []
I20260927 23:31:00.307328 3117 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.324533 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.331959 1730 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:00.332444 2285 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.320077 3075 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:00.333123 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1120) took 1386 ms. Trace:
I20260927 23:31:00.387197 3145 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.391852 1584 rpcz_store.cc:276] 0927 23:30:58.946896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.957043 (+ 10147us) service_pool.cc:227] Handling call
0927 23:30:59.029896 (+ 72853us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.220724 (+190828us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:30:59.360012 (+139288us) log.cc:842] Serialized 14 byte log entry
0927 23:30:59.423511 (+ 63499us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:30:59.426034 (+ 2523us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:30:59.429219 (+ 3185us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:30:59.930417 (+501198us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:00.327502 (+397085us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54001fead4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:00.330080 (+ 2578us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:00.333070 (+ 2990us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5591,"prepare.queue_time_us":156940,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":14883,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":3865,"spinlock_wait_cycles":394987648,"thread_start_us":136665,"threads_started":2,"wal-append.queue_time_us":84603}
I20260927 23:31:00.333294 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.394646 2288 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:00.394994 3194 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.395443 3194 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.397344 3194 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.318017 3192 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260927 23:31:00.334360 3165 raft_consensus.cc:3032] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.378960 1655 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.337816 1579 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:00.368578 1587 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.368398 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:00.269191 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:00.370532 3202 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260927 23:31:00.371371 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
W20260927 23:31:00.371428 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 55502.4us
I20260927 23:31:00.404654 1554 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.404772 1551 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.371971 1657 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.316550 3199 consensus_queue.cc:1046] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260927 23:31:00.407568 1657 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:00.374495 1661 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.310232 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:46200 (request call id 1307) took 1767 ms. Trace:
I20260927 23:31:00.380548 1737 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.382040 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:00.307175 2906 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260927 23:31:00.302067 3183 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:00.389519 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 72740.4us
I20260927 23:31:00.413067 1700 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.388217 3164 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.408319 1663 rpcz_store.cc:276] 0927 23:30:58.535767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.543348 (+ 7581us) service_pool.cc:227] Handling call
0927 23:30:59.105145 (+561797us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c8030 00007f97559b33f9 00007f97559b2e95 00007f974f0a4784
0927 23:31:00.248813 (+1143668us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540013b0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a66da7 00007f97559b2ee0 00007f974f0a4784 00007f974f0a4712 00007f974f0a4641 00007f974f0a45bc 00007f974f0a3872 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:00.303212 (+ 54399us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572715264}
I20260927 23:31:00.390892 1627 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.415405 1627 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.393942 3145 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.334642 2286 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:00.334828 1658 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:00.320653 3125 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.401031 2291 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.409626 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.418072 1735 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:00.419395 2293 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.401438 3165 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.438731 3199 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:00.390000 3152 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.439155 3199 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:00.405665 3184 consensus_queue.cc:237] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.439550 3199 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:00.439829 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.381124 3052 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260927 23:31:00.457798 1551 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.380604 3102 raft_consensus.cc:686] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:00.420672 1627 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.421716 1554 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:00.465315 1554 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:00.446439 3181 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.422137 2258 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:00.424494 3183 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.425976 3014 raft_consensus.cc:3032] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.426357 2906 raft_consensus.cc:3032] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.426733 3134 raft_consensus.cc:3032] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.428650 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.510226 1728 raft_consensus.cc:2379] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:00.510475 3209 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.478811 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 139) took 1373 ms. Trace:
I20260927 23:31:00.511262 1732 rpcz_store.cc:276] 0927 23:30:59.105632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:59.261573 (+155941us) service_pool.cc:227] Handling call
0927 23:30:59.261865 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:00.094609 (+832744us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:00.456276 (+361667us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b540024f9d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:00.476181 (+ 19905us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1188056960}
I20260927 23:31:00.511300 3061 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.512064 1737 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.431203 3143 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.432345 3075 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.401787 2938 consensus_queue.cc:1046] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260927 23:31:00.417928 3120 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.409257 3192 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.434113 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.435300 3137 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.435616 3153 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.436771 3126 raft_consensus.cc:3032] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.436898 2287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 378) took 1653 ms. Trace:
W20260927 23:31:00.438144 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56157.6us
I20260927 23:31:00.389777 2289 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.515197 3134 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:00.515905 3052 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.403746 3108 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.442030 1485 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 148) took 4058 ms. Trace:
I20260927 23:31:00.443534 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.383808 3146 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.450388 2257 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.452436 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1130) took 1471 ms. Trace:
I20260927 23:31:00.418427 3163 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.474409 3182 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
W20260927 23:31:00.534884 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 91451.5us
I20260927 23:31:00.477265 1627 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:00.485487 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 63429.6us
I20260927 23:31:00.536060 1700 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.508347 3040 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:00.508447 3198 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.508955 1551 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.429004 3125 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.485061 3166 consensus_queue.cc:1046] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260927 23:31:00.512871 3061 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.490085 2906 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.490242 3014 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.548396 3165 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.513451 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.429478 3015 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:00.461273 3072 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.466993 3174 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.478180 2901 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:00.521096 1484 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 1 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:00.521922 3162 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.533140 1628 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:00.533653 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 83407.2us
I20260927 23:31:00.531924 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.567027 1737 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:00.568099 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:00.534332 1659 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.475286 3193 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.518257 3109 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.532938 1580 rpcz_store.cc:276] 0927 23:30:58.981144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.981413 (+ 269us) service_pool.cc:227] Handling call
0927 23:30:59.002210 (+ 20797us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.657310 (+655100us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:00.452180 (+794870us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b5400467154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:00.452372 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1499480832}
I20260927 23:31:00.538767 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:00.539463 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 62306.8us
I20260927 23:31:00.540300 1700 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.575821 1627 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.576272 1700 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.576980 1627 leader_election.cc:398] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:00.577270 1627 leader_election.cc:401] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:31:00.577304 1700 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.533196 1485 rpcz_store.cc:276] 0927 23:30:56.383422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:56.383605 (+ 183us) service_pool.cc:227] Handling call
0927 23:31:00.441974 (+4058369us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:30:58.459131 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:30:58.522361 (+ 63230us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:30:58.522392 (+ 31us) write_op.cc:435] Acquired schema lock
0927 23:30:58.522412 (+ 20us) tablet.cc:660] Decoding operations
0927 23:30:58.523022 (+ 610us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:30:58.523092 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0927 23:30:58.523117 (+ 25us) tablet.cc:683] Acquiring locks for 4 operations
0927 23:30:58.523488 (+ 371us) tablet.cc:699] Row locks acquired
0927 23:30:58.523505 (+ 17us) write_op.cc:260] PREPARE: finished
0927 23:30:58.523637 (+ 132us) write_op.cc:270] Start()
0927 23:30:58.523728 (+ 91us) write_op.cc:276] Timestamp: P: 1790551858523596 usec, L: 0
0927 23:30:58.523746 (+ 18us) op_driver.cc:331] REPLICATION: starting
0927 23:30:58.580902 (+ 57156us) log.cc:842] Serialized 1770 byte log entry
0927 23:30:58.807249 (+226347us) op_driver.cc:444] REPLICATION: finished
0927 23:30:59.109041 (+301792us) write_op.cc:301] APPLY: starting
0927 23:30:59.116055 (+ 7014us) tablet.cc:1365] starting BulkCheckPresence
0927 23:30:59.165867 (+ 49812us) tablet.cc:1368] finished BulkCheckPresence
0927 23:30:59.179267 (+ 13400us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:30:59.276699 (+ 97432us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:30:59.276873 (+ 174us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0927 23:30:59.276942 (+ 69us) write_op.cc:312] APPLY: finished
0927 23:30:59.325813 (+ 48871us) log.cc:842] Serialized 41 byte log entry
0927 23:30:59.343177 (+ 17364us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:59.359590 (+ 16413us) write_op.cc:454] Released schema lock
0927 23:30:59.361853 (+ 2263us) write_op.cc:341] FINISH: Updating metrics
0927 23:30:59.378832 (+ 16979us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:30:59.383256 (+ 4424us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":21463,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":258361,"apply.run_cpu_time_us":4201,"apply.run_wall_time_us":294362,"num_ops":4,"prepare.queue_time_us":125712,"prepare.run_cpu_time_us":3948,"prepare.run_wall_time_us":162312,"replication_time_us":261149,"thread_start_us":276289,"threads_started":2}]]}
I20260927 23:31:00.578408 1700 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.546666 3181 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.546825 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.547112 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.548013 1584 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:00.537966 3040 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:00.526450 3126 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.537381 3128 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:31:00.571816 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:00.542251 3204 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.546594 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.589464 1732 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.542834 3192 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.573937 2293 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:00.580672 1627 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.550493 3084 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:00.554376 3153 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.582480 3209 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.546594 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.553328 3182 raft_consensus.cc:686] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.588335 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.542867 3108 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.563925 2901 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:00.551570 3145 consensus_queue.cc:1046] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260927 23:31:00.574972 3164 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:00.655303 3104 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.578439 3148 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:00.659446 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 78830.2us
I20260927 23:31:00.549338 3014 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:00.573850 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 60502.4us
I20260927 23:31:00.534569 3052 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.695973 1654 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:00.696362 3182 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.697029 3052 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.698220 2293 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:00.700583 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:00.701030 3171 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.555670 3102 consensus_queue.cc:237] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.702489 1660 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:00.702396 3052 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:00.662972 1585 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.591261 1484 catalog_manager.cc:5495] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 3, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:00.585255 2910 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.580247 3189 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.580072 3066 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:00.526145 2287 rpcz_store.cc:276] 0927 23:30:58.784026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.801258 (+ 17232us) service_pool.cc:227] Handling call
0927 23:30:58.832064 (+ 30806us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:30:59.772713 (+940649us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:00.436633 (+663920us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b540036d3d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:00.436819 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1994883328}
I20260927 23:31:00.668942 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:00.555958 3198 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.579916 3152 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.659482 3181 consensus_queue.cc:1046] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
W20260927 23:31:00.672780 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 84556.6us
I20260927 23:31:00.730774 1700 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:00.665030 3192 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.644656 3175 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.590674 3174 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:00.594091 3166 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.656082 3128 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.673940 1655 raft_consensus.cc:2379] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:00.592100 3202 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.686846 3206 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.686925 3125 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.687561 1581 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:00.690486 3207 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.690845 3191 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
W20260927 23:31:00.692083 3210 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260927 23:31:00.517866 2290 raft_consensus.cc:2932] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:00.666785 3176 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.674906 2901 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.594318 3167 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.672034 3118 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.711623 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.673978 3209 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 23:31:00.714437 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45575.5us
I20260927 23:31:00.715260 1658 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:00.715369 3015 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:00.716053 3108 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.664719 2905 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.709609 3010 raft_consensus.cc:3032] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.719297 3014 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.583773 3040 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:00.583895 3074 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.723948 3066 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:00.721606 3164 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.583566 3134 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.707706 3104 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.741442 3203 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:00.678268 3126 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.770680 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:00.770885 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 49416.2us
I20260927 23:31:00.745117 3072 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:00.747423 3146 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.748822 3206 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.721212 3214 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260927 23:31:00.734743 3163 consensus_queue.cc:1046] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260927 23:31:00.790254 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:00.790710 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 60049us
I20260927 23:31:00.826414 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:00.749399 3125 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.874629 1737 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.876439 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:00.878132 1585 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:00.879524 1588 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:00.861889 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.863711 2901 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.768745 3207 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.788556 3177 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260927 23:31:00.788491 3108 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.889582 2257 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.863711 3193 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:00.785502 3015 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:00.789951 3066 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:00.905109 3166 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.815896 1732 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.867846 3194 raft_consensus.cc:3032] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:00.869987 1728 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.818248 3198 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.787781 3010 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:00.819257 3192 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.781243 3171 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.881305 3096 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.862764 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.921444 1656 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:00.807246 1583 raft_consensus.cc:1261] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:00.908380 1731 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:00.861580 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.910960 3146 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.923589 3015 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.913883 2910 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.922974 3184 raft_consensus.cc:3032] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:00.925441 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:00.938083 3040 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.940303 2289 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:31:00.951933 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 66115.4us
I20260927 23:31:00.818526 3128 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.913712 3143 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:00.829959 3118 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.889461 3072 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.964556 3192 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:00.992462 1629 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:00.816731 3199 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.970256 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:00.995313 2291 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:00.951004 3165 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.970310 1730 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:00.891543 2906 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.933816 3194 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:00.984516 1734 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:01.018957 1733 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:00.989244 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:00.931545 3202 consensus_queue.cc:1046] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260927 23:31:00.968837 3146 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.033037 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:00.948974 3010 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.870222 3164 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.997078 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:00.971704 3184 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:00.970682 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:00.970409 2286 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:00.935561 3166 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.996912 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.044067 3226 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:01.036193 3145 consensus_queue.cc:1046] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260927 23:31:00.997484 3177 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:00.997329 3192 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.063952 3177 raft_consensus.cc:686] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:01.065428 3146 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.065094 3177 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.066903 3166 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.003115 3125 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.003235 3118 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.988204 3143 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.044211 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.069393 3143 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.005738 1730 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:01.006724 3104 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:00.985208 3108 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:01.009732 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.071653 2287 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:01.010874 2901 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.011880 3015 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.015476 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:00.985450 3171 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:00.949030 3096 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:00.934470 3074 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260927 23:31:01.030719 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:01.031616 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.032356 1729 raft_consensus.cc:3032] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:01.090492 1729 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:01.034662 1737 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:01.054921 3010 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:00.976768 2910 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:00.988991 3175 consensus_queue.cc:1046] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260927 23:31:01.098618 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.102046 3106 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:01.091215 3171 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.102813 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:01.037710 1656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1424) took 1534 ms (client timeout 1771 ms). Trace:
W20260927 23:31:01.103482 1656 rpcz_store.cc:269] 0927 23:30:59.500170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:59.522078 (+ 21908us) service_pool.cc:227] Handling call
0927 23:31:01.032690 (+1510612us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540013a6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:01.032784 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2600537472}
I20260927 23:31:01.104001 1728 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:01.018828 3193 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.104401 3171 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.037436 3163 consensus_queue.cc:1035] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260927 23:31:01.108489 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:01.052279 3040 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.052932 3072 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.109225 1656 raft_consensus.cc:2379] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:01.049908 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.109615 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.056113 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.058894 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.110204 1658 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:01.111166 1735 raft_consensus.cc:2379] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:01.111994 2287 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:01.053319 3184 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.061668 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:01.062136 3199 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:00.978636 3052 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:01.071732 3126 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.010457 3207 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:01.076063 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.076296 1703 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.035236 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.087446 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:01.093667 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 63019.6us
I20260927 23:31:01.123905 1551 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:01.123942 1588 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:31:01.037425 3061 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260927 23:31:01.072777 3206 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.098939 3015 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.098595 1729 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.100521 2910 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.102828 3106 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:01.039801 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.062490 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.040854 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.104616 3193 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.053594 3167 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.118984 1659 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:01.091811 1655 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:01.120321 1554 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:01.094678 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:01.127319 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 52520.7us
I20260927 23:31:01.129914 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:01.121825 3066 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.105412 3120 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election won for term 2
W20260927 23:31:01.143893 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 108567us
I20260927 23:31:01.132215 3199 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.147863 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:01.148365 1551 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.134177 3074 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:01.134051 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.160077 1734 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:01.144598 3015 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.128639 3052 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.189162 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.190007 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.190917 1734 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:31:01.192062 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.192812 1588 raft_consensus.cc:1261] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:01.192986 1734 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:01.194119 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.195055 1734 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:01.195308 1588 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:01.197129 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.199093 1588 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 23:31:01.200156 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 51857.5us
I20260927 23:31:01.203466 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.148839 2910 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.148013 3184 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.160275 3206 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.213608 1734 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:01.165535 2938 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:01.213549 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.164115 3176 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:31:01.222293 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.223037 3102 raft_consensus.cc:3032] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:01.160929 3199 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.175972 3096 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.175765 2905 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.224543 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.159934 3074 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.201676 3153 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.221035 2910 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:01.201567 3072 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:01.164577 3120 raft_consensus.cc:686] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:01.254051 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.228014 1737 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:01.207158 3054 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:01.210956 3108 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:01.205152 3016 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.258831 1658 raft_consensus.cc:1261] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:01.236951 1728 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:01.259089 3016 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.259553 3016 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.233915 1661 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:01.208133 3224 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.234920 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.243183 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.229276 2906 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.246776 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.232959 3102 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:01.202895 3015 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:01.252737 2910 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.277690 1700 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.233085 3128 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.258136 2901 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:01.258175 3054 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.262107 1627 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.233215 3098 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.265259 1733 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:01.278925 3075 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:01.233307 2938 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:01.242035 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.270160 2258 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.257346 3074 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:01.261435 3108 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:01.257176 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.208662 3106 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.315734 1730 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:01.316673 3125 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:31:01.317284 3125 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:01.318408 3125 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.320271 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:01.321974 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.322551 3146 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.323088 1731 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:01.323194 3146 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.324887 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.325686 1732 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.325740 3146 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:01.328662 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 71554.4us
I20260927 23:31:01.330273 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.330976 3148 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:01.331169 3146 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:01.331634 3148 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.333235 1552 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.336311 3146 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:01.336766 3146 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:01.337198 3146 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.337670 3193 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.338158 2293 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 3. Ops: []
I20260927 23:31:01.281044 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.343766 3194 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.344969 3194 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.345049 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.236145 3199 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.286613 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:01.344414 3148 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.349746 3148 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:01.279031 2901 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:01.354110 2901 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.355839 3098 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:01.356258 3098 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.350078 1579 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:01.350440 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.359298 1579 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:01.359709 3163 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.360214 3163 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.361552 1729 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.359400 1583 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:01.362854 3163 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:01.363940 3125 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.364310 3125 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.364606 3125 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.363391 3163 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260927 23:31:01.257908 3162 consensus_meta.cc:287] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.685s user 0.005s sys 0.006s
I20260927 23:31:01.367992 3148 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.368855 3134 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.371927 3163 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:01.275262 3015 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:01.273507 3202 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.374943 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.377647 1729 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:01.357041 3146 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.251613 3164 consensus_queue.cc:1035] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 23:31:01.280484 3097 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.253978 3153 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.300649 3096 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:01.253532 3171 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 23:31:01.269629 3224 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.289587 3072 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.309568 3108 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:01.414700 3148 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.309307 3166 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:01.394310 3096 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.284837 3219 consensus_queue.cc:1035] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.112s
I20260927 23:31:01.416220 3096 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.398180 3146 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.403313 3192 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.403868 2938 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.407258 3054 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.348776 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.422108 1731 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:01.423138 3096 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.424008 1552 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.424377 3192 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.424815 3146 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:01.424814 3192 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.425330 3146 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.425725 3146 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.426415 3096 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.375826 2288 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:01.431628 1662 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 23:31:01.278873 3128 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.432817 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.435881 1737 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.301935 3152 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
W20260927 23:31:01.391745 3209 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260927 23:31:01.407361 3016 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.457994 2293 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:01.429019 3106 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.469832 1737 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:01.471464 1554 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:01.448855 3228 log.cc:924] Time spent T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
I20260927 23:31:01.432726 3192 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.473451 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.476830 3061 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:01.392099 3097 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.434230 3066 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.447952 3219 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.473603 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.450986 3131 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.457787 3052 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.495618 3193 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 23:31:01.428664 3072 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.392103 3202 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.496239 3118 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:01.515889 3221 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.311s user 0.001s sys 0.000s
I20260927 23:31:01.493224 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.433861 3120 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.532982 1736 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:01.540522 3074 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.541625 3072 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.542299 3016 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.518035 3061 raft_consensus.cc:686] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:01.559731 3232 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.335s user 0.000s sys 0.004s
I20260927 23:31:01.559842 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:01.572316 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.497129 1585 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:01.519430 3131 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.545275 3118 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:01.557878 1728 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
W20260927 23:31:01.578630 3231 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.354s user 0.000s sys 0.002s
I20260927 23:31:01.591722 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.596716 3153 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.581116 2901 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.577792 3214 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
W20260927 23:31:01.602751 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 313870us
I20260927 23:31:01.610744 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:01.614389 2293 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:01.535209 1731 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.615703 1627 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.616039 1554 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.614837 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.617177 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.620189 2292 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:01.623502 1627 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.525302 3219 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.626241 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.619105 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:01.608338 1732 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:01.634003 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.680634 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.702193 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:01.709197 2292 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:01.663125 1554 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:01.697744 1586 raft_consensus.cc:595] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:01.710197 1586 raft_consensus.cc:603] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:01.703383 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:01.711249 1579 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:01.711766 1585 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:31:01.709380 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 75454.5us
I20260927 23:31:01.714380 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.617376 3153 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.618021 3224 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.737044 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:01.677659 3241 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260927 23:31:01.857179 1662 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 23:31:01.858006 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 143254us
I20260927 23:31:01.883558 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.885205 1627 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.632242 3226 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.631938 2938 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:01.627424 3193 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:01.662009 3165 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:01.875237 1581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.033s sys 0.086s
W20260927 23:31:01.953301 1581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s user 0.033s sys 0.086s
I20260927 23:31:01.875465 1729 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:01.879987 2258 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:01.881202 1629 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.007s sys 0.033s
W20260927 23:31:01.862864 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 199857us
I20260927 23:31:01.653066 3016 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.674134 2901 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.691435 3074 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:01.679401 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:01.634276 3052 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:01.678952 2290 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 356) took 1287 ms. Trace:
I20260927 23:31:01.664337 3152 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:01.939466 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:01.942458 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57319.3us
I20260927 23:31:01.944352 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:01.890034 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:01.881042 3126 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.735909 1728 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:01.938748 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:01.946885 3135 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.949891 1662 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 23:31:02.003090 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 123209us
W20260927 23:31:02.003669 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59422.2us
I20260927 23:31:01.714475 3097 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:01.715045 3148 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.004925 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.005687 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.005823 1737 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.006243 2289 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:01.760075 2905 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.004842 3148 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.007165 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.006948 2905 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.008088 1703 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.008821 3148 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.009367 2905 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:01.719960 3174 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:01.701910 3118 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.736300 3040 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:01.981309 1629 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.012s sys 0.037s
I20260927 23:31:01.959475 1663 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:01.678156 3102 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:01.875744 3194 consensus_meta.cc:287] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.002s
I20260927 23:31:01.883044 3072 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:31:01.997300 3146 consensus_meta.cc:287] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.000s
W20260927 23:31:01.951545 3098 consensus_meta.cc:287] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.595s user 0.011s sys 0.001s
I20260927 23:31:01.867319 3066 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.634550 3164 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:01.931855 3163 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.932351 3054 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:01.999908 1661 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:01.715284 3143 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:01.984073 2286 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.116s sys 0.163s
I20260927 23:31:01.882102 3014 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260927 23:31:01.930958 3131 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.012899 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:01.690470 3061 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.991991 1660 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:02.017298 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:01.986212 2938 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:01.946197 3226 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.866554 3104 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.021647 3137 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:01.984330 3165 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.012902 3128 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.018460 3040 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:01.953514 3166 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260927 23:31:01.997663 3204 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:01.933260 3153 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:02.011386 2906 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.731s user 0.009s sys 0.001s
I20260927 23:31:02.048856 3163 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.979549 3175 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 23:31:02.030159 2292 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:02.020608 3235 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.500s user 0.000s sys 0.001s
I20260927 23:31:02.099452 1703 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:01.995754 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1342) took 1030 ms. Trace:
I20260927 23:31:02.015542 3109 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:02.045730 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 9) took 1670 ms. Trace:
I20260927 23:31:01.999596 1584 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:02.019057 3072 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:02.088970 2910 consensus_meta.cc:287] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.787s user 0.008s sys 0.000s
I20260927 23:31:02.104256 1586 rpcz_store.cc:276] 0927 23:31:00.364520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:00.371241 (+ 6721us) service_pool.cc:227] Handling call
0927 23:31:02.034842 (+1663601us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400061154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a69bc3 00007f97559b381a 00007f974f0a6d04 00007f974f0a6c92 00007f974f0a6bc1 00007f974f0a6b3c 00007f974f0a5df2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:02.034981 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2975188352}
I20260927 23:31:02.022505 3097 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.004232 3126 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:02.105615 3108 consensus_meta.cc:287] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.657s user 0.010s sys 0.000s
I20260927 23:31:01.986742 3016 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.032294 3146 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:01.995232 3152 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.106411 3175 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:02.070250 3125 consensus_meta.cc:287] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.693s user 0.010s sys 0.000s
I20260927 23:31:02.090652 3202 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.078296 3226 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:02.120672 3167 consensus_meta.cc:287] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.979s user 0.012s sys 0.000s
I20260927 23:31:02.137553 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.138134 1628 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.138777 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.139539 1579 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.140555 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.140779 1628 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.086889 3040 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.077982 3214 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.094589 1659 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
W20260927 23:31:02.049968 2286 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.827s user 0.116s sys 0.164s
I20260927 23:31:02.095923 1729 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:02.096956 1661 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260927 23:31:02.081516 3234 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.004s sys 0.023s
I20260927 23:31:02.193053 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.193531 1552 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.194419 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.014150 3135 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.019770 2905 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.038005 3066 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.078186 3137 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:02.051721 3224 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260927 23:31:02.045382 3164 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.092478 1728 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:02.113974 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.141245 1579 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.213420 1551 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.148841 3214 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:02.091745 3153 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.092952 3165 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.117940 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.110997 3176 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:02.113530 3109 raft_consensus.cc:686] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:02.050472 3143 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.091812 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.111184 3072 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.108603 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.054183 1737 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
W20260927 23:31:02.080442 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 133192us
I20260927 23:31:02.111799 3097 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:02.119868 3199 consensus_meta.cc:287] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.773s user 0.010s sys 0.000s
W20260927 23:31:02.120919 3206 consensus_meta.cc:287] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.891s user 0.006s sys 0.005s
I20260927 23:31:02.080039 3104 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.246624 3252 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:02.247082 2285 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:02.103787 1654 rpcz_store.cc:276] 0927 23:31:00.964977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:00.968346 (+ 3369us) service_pool.cc:227] Handling call
0927 23:31:00.971017 (+ 2671us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:01.066634 (+ 95617us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:01.066664 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:01.067097 (+ 433us) log.cc:842] Serialized 26 byte log entry
0927 23:31:01.150424 (+ 83327us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:01.161287 (+ 10863us) log.cc:842] Serialized 14 byte log entry
0927 23:31:01.165094 (+ 3807us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:01.360032 (+194938us) spinlock_profiling.cc:243] Waited 95.3 ms on lock 0x7b54001cfcd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:01.360076 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:01.889503 (+529427us) raft_consensus.cc:1671] finished
0927 23:31:01.979985 (+ 90482us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:01.992322 (+ 12337us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3515,"prepare.queue_time_us":378,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":4307,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":238,"spinlock_wait_cycles":219096448,"thread_start_us":70574,"threads_started":1,"wal-append.queue_time_us":77516}
I20260927 23:31:02.248416 3104 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.090294 3204 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.249226 3146 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.235879 3253 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.250125 3253 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:01.991972 2290 rpcz_store.cc:276] 0927 23:31:00.391363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:00.391507 (+ 144us) service_pool.cc:227] Handling call
0927 23:31:00.420017 (+ 28510us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:00.490588 (+ 70571us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:01.068700 (+578112us) log.cc:842] Serialized 14 byte log entry
0927 23:31:01.115094 (+ 46394us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:01.206330 (+ 91236us) log.cc:842] Serialized 26 byte log entry
0927 23:31:01.223842 (+ 17512us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:01.226031 (+ 2189us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:01.231071 (+ 5040us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:01.490865 (+259794us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:01.678555 (+187690us) raft_consensus.cc:1671] finished
0927 23:31:01.678696 (+ 141us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:01.678866 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16942,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":1946,"spinlock_wait_cycles":4502784,"thread_start_us":26144,"threads_started":1,"wal-append.queue_time_us":77875}
I20260927 23:31:02.250494 3253 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.250586 3204 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.238288 2289 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:02.110775 3219 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.179461 3166 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.223279 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.251926 1658 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.251801 3166 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.252902 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.221107 1730 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:02.180943 2938 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:02.254675 2938 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:02.256939 1731 consensus_meta.cc:287] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.638s user 0.004s sys 0.004s
I20260927 23:31:02.257282 1731 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:02.257793 2938 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.181025 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.258262 3126 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.258493 2292 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:02.181592 3247 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.259749 3247 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.125856 3163 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.183492 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.261456 1662 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:02.220081 2901 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:02.182060 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.113250 2910 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.178056 3061 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.130718 3125 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.189268 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.273487 1663 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.224677 3176 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:02.121272 3108 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.177361 3174 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.209304 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.146193 3171 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:02.213727 1586 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:02.039512 3098 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.214483 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.214581 1628 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.175405 3167 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:02.196157 3234 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.004s sys 0.024s
I20260927 23:31:02.186743 2906 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.221820 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:02.224857 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 107035us
I20260927 23:31:02.196822 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.233933 2294 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:02.211767 3137 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.196131 2905 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.207928 3128 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:02.238108 1552 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.219486 3214 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.245155 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:02.242057 1732 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:02.268373 3194 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.299733 1587 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:02.303165 3247 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.302867 3194 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.304018 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 172) took 1023 ms. Trace:
I20260927 23:31:02.275787 3061 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.304348 1587 rpcz_store.cc:276] 0927 23:31:01.280624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.280785 (+ 161us) service_pool.cc:227] Handling call
0927 23:31:02.303870 (+1023085us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b5400466c54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:02.303967 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2199030144}
I20260927 23:31:02.305388 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.254186 3166 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.305809 3194 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.242686 3097 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.261368 3199 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:02.307154 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 62120.1us
I20260927 23:31:02.307355 2905 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.307935 3199 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:02.290277 3246 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260927 23:31:02.310559 3075 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.270747 3014 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:02.311443 3014 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:02.289997 3236 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
I20260927 23:31:02.313616 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1349) took 1065 ms. Trace:
I20260927 23:31:02.313769 3014 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.262694 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.261162 3117 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.314209 3174 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:02.314544 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 51924.6us
I20260927 23:31:02.314499 2906 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.263614 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.313970 1656 rpcz_store.cc:276] 0927 23:31:01.248549 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.248915 (+ 366us) service_pool.cc:227] Handling call
0927 23:31:01.249308 (+ 393us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:01.256026 (+ 6718us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:01.256053 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:01.258717 (+ 2664us) log.cc:842] Serialized 26 byte log entry
0927 23:31:01.574040 (+315323us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:01.574079 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:01.574163 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:02.313422 (+739259us) raft_consensus.cc:1671] finished
0927 23:31:02.313440 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:02.313548 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7454,"raft.queue_time_us":12002,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":4360,"thread_start_us":314982,"threads_started":1,"wal-append.queue_time_us":755295}
I20260927 23:31:02.316386 2906 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.316764 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.317332 1656 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
W20260927 23:31:02.317552 3184 consensus_meta.cc:287] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.093s user 0.007s sys 0.004s
I20260927 23:31:02.266960 2293 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.293579 1585 raft_consensus.cc:2354] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:02.270147 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.318779 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
W20260927 23:31:02.288070 3257 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260927 23:31:02.144343 3175 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:02.319485 1586 raft_consensus.cc:595] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:02.319924 1586 raft_consensus.cc:603] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 23:31:02.280368 3242 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260927 23:31:02.321923 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1287) took 1121 ms. Trace:
I20260927 23:31:02.322646 3098 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:02.322829 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 108548us
I20260927 23:31:02.323498 3164 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.323714 3052 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.290864 3176 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.324119 3052 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.324479 3052 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.324520 3176 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:02.325161 3258 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
W20260927 23:31:02.272037 3106 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.793s user 0.007s sys 0.004s
I20260927 23:31:02.326835 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.327199 2291 raft_consensus.cc:2379] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:02.311669 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.294960 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46514 (request call id 853) took 1422 ms. Trace:
I20260927 23:31:02.310953 3075 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:02.239179 3134 consensus_meta.cc:287] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.870s user 0.009s sys 0.003s
I20260927 23:31:02.318507 1734 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:02.330072 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.341331 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.329867 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:02.236094 3193 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.341578 2293 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.320386 3175 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.293156 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.352348 1663 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.328027 1582 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:02.248797 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:02.316380 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 52837.1us
I20260927 23:31:02.305989 1587 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.238688 3256 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:02.269145 2258 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:02.364826 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 95775.8us
I20260927 23:31:02.318152 3108 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.314705 3174 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.327395 2292 raft_consensus.cc:1799] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Handling vote request from an unknown peer 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:02.300607 2910 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.334934 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.337819 1735 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:02.318913 1662 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.342013 3075 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.342698 2291 raft_consensus.cc:2354] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:02.374691 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.375768 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.343235 1656 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:02.328223 1580 rpcz_store.cc:276] 0927 23:31:00.872507 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:00.872669 (+ 162us) service_pool.cc:227] Handling call
0927 23:31:02.294418 (+1421749us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400308ed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4af0 00007f9750a77f2a 00007f97559b4c6e 00007f974f0ab807 00007f974f0ab792 00007f974f0ab6c1 00007f974f0ab63c 00007f974f0aa8f2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:02.294811 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2782372736}
I20260927 23:31:02.345714 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.377053 1659 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:02.377812 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:02.346094 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 108067us
I20260927 23:31:02.379837 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.381280 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.348109 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.382624 1661 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
W20260927 23:31:02.270666 3015 consensus_meta.cc:287] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.881s user 0.011s sys 0.000s
I20260927 23:31:02.322260 1581 rpcz_store.cc:276] 0927 23:31:01.200493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.201225 (+ 732us) service_pool.cc:227] Handling call
0927 23:31:01.201482 (+ 257us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:01.201813 (+ 331us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:01.201852 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:01.209154 (+ 7302us) log.cc:842] Serialized 26 byte log entry
0927 23:31:01.953661 (+744507us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:01.953700 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:01.953763 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:02.321751 (+367988us) raft_consensus.cc:1671] finished
0927 23:31:02.321768 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:02.321874 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6707,"thread_start_us":720062,"threads_started":1,"wal-append.queue_time_us":904099}
I20260927 23:31:02.383656 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.384217 1581 raft_consensus.cc:1799] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Handling vote request from an unknown peer 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:02.326157 3040 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:02.291181 3249 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.266s user 0.002s sys 0.000s
I20260927 23:31:02.318120 3184 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:31:02.285935 3251 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.261s user 0.000s sys 0.001s
I20260927 23:31:02.354028 2259 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:02.296953 3244 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.270s user 0.000s sys 0.002s
I20260927 23:31:02.351573 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:02.361768 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.285110 3066 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.322841 3143 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.370587 2292 raft_consensus.cc:2379] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:02.364651 2287 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:02.283790 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.366266 3125 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.305812 3061 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.371109 2293 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:02.373422 3194 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.284269 3016 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.470763 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.291821 3152 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.376307 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.382675 1663 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:02.384740 1581 raft_consensus.cc:2396] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260927 23:31:02.289110 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.387063 1628 leader_election.cc:398] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:02.474989 1628 leader_election.cc:401] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:02.387665 1700 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.350108 3128 raft_consensus.cc:686] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:02.349903 3126 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:02.478006 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.398870 2257 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:02.478955 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 80188.3us
I20260927 23:31:02.389205 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.389694 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.388247 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1159) took 1152 ms. Trace:
I20260927 23:31:02.480309 1662 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.480571 1734 rpcz_store.cc:276] 0927 23:31:01.237442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.238393 (+ 951us) service_pool.cc:227] Handling call
0927 23:31:01.238726 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:02.341228 (+1102502us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:02.388042 (+ 46814us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001d1354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:02.388196 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2482881408}
W20260927 23:31:02.408655 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 54716.9us
W20260927 23:31:02.481352 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 94425us
I20260927 23:31:02.389055 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.416833 1580 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:02.365957 3209 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.314586 3117 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.341611 3167 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.459192 1585 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:02.459375 2294 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:02.460546 1733 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:02.487092 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:02.417629 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 66166.8us
W20260927 23:31:02.418138 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 56482.9us
I20260927 23:31:02.453729 1629 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.490240 1734 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.491252 1551 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:02.491858 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.492406 1734 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.379068 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 304) took 1192 ms. Trace:
I20260927 23:31:02.465301 3224 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.355530 3193 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.472748 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.283121 3109 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.407014 3040 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.475332 1737 raft_consensus.cc:2379] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.476840 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.511947 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.512315 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.512383 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.494216 3204 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.513038 1737 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.402455 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.503723 3177 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:02.513640 1661 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 1.
I20260927 23:31:02.513264 3204 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.482187 1659 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.514191 3204 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.514554 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.514886 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:02.484714 3262 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.485069 3091 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:02.354768 3199 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.480361 1656 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.408548 3184 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:02.485437 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 97927.4us
I20260927 23:31:02.487637 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.494863 3145 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:02.326745 3171 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:02.371265 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.406872 3014 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260927 23:31:02.474743 2910 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.402616 3015 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.498500 1554 leader_election.cc:398] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:02.540982 1554 leader_election.cc:401] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260927 23:31:02.498591 1551 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.500684 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.408308 3146 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260927 23:31:02.373422 3256 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.457881 3166 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260927 23:31:02.532634 3243 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.501297 3016 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260927 23:31:02.515143 1661 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.536521 3171 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:02.488684 3209 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.415961 3134 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.515956 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.465512 3247 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260927 23:31:02.458082 3253 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.489436 3206 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.519747 1736 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.513242 1660 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.411871 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:02.523743 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 126691us
I20260927 23:31:02.494637 3102 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:02.388069 3137 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.529371 3239 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260927 23:31:02.548312 3061 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:02.571355 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 182724us
I20260927 23:31:02.531391 1580 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.570256 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.547858 3125 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.518549 3040 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.495734 3135 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.533675 1737 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.500119 3224 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:02.543311 2910 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.474380 3194 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.485183 3126 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.585706 3126 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.580199 3224 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.541384 1656 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.541754 1734 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.545189 3128 consensus_queue.cc:237] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.592705 1734 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
W20260927 23:31:02.542273 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 43769.9us
I20260927 23:31:02.593518 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.593602 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.594135 1734 raft_consensus.cc:2354] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:02.575026 3143 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.594892 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:02.544420 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 46083.9us
I20260927 23:31:02.530880 3199 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.531051 3074 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.536541 3184 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.549274 1661 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 23:31:02.388293 3098 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:02.559973 1588 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:02.532677 3091 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.569022 1663 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:31:02.566943 3243 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.529995 3262 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.551642 1730 raft_consensus.cc:2379] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.545200 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1292) took 1170 ms. Trace:
I20260927 23:31:02.387687 3108 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.531394 1700 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.572055 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.494580 3114 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:02.609026 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 77715.7us
I20260927 23:31:02.596576 3126 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:02.599033 3199 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.574656 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.562168 3134 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.576923 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.613524 1737 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.561070 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.504678 3054 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:02.616320 3248 log.cc:924] Time spent T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.522s user 0.002s sys 0.002s
I20260927 23:31:02.616585 3054 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.617030 3091 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.617515 3126 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:02.617271 2905 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.618172 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.618508 3106 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:02.618808 1730 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.619081 3106 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.595527 1734 raft_consensus.cc:2354] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:02.575246 3066 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260927 23:31:02.573891 1735 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.609668 1700 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.572387 1552 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.597281 3143 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.612854 1580 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:02.615190 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.608340 3114 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.608632 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.602267 3074 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.639539 3091 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:02.629937 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57596.9us
I20260927 23:31:02.647905 3276 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.648029 3247 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:02.648612 3276 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.649159 3276 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.651422 1628 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.652284 3247 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.652680 3247 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.653074 3247 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.629217 3207 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:02.609989 3193 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.635773 3146 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.629230 3104 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.638602 1728 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.568591 3226 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.629832 1581 raft_consensus.cc:2354] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:02.612571 2285 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:02.640659 1662 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:02.640722 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.643544 2293 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:02.630467 1737 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.687646 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.688242 1737 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.689284 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.589646 3097 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:02.686000 3268 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260927 23:31:02.691112 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.691915 3226 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.692349 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.693583 1586 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 23:31:02.630504 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:02.656574 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 119699us
I20260927 23:31:02.614441 3134 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.607322 3167 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.494499 2286 rpcz_store.cc:276] 0927 23:31:01.180752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.182782 (+ 2030us) service_pool.cc:227] Handling call
0927 23:31:01.206129 (+ 23347us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:01.213285 (+ 7156us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:02.219373 (+1006088us) log.cc:842] Serialized 14 byte log entry
0927 23:31:02.355858 (+136485us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:02.355930 (+ 72us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:02.370030 (+ 14100us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:02.373186 (+ 3156us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:02.373319 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15159,"prepare.queue_time_us":78799,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":4775,"raft.queue_time_us":2661,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":9365,"spinlock_wait_cycles":15516416,"thread_start_us":1040756,"threads_started":2,"wal-append.queue_time_us":177829}
I20260927 23:31:02.639818 3214 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.707727 3040 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.618405 3126 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.672174 3269 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:02.719334 3072 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.685339 3194 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.687754 3152 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:02.678491 3262 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.641243 3143 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.678493 3209 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.682542 3106 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.698547 1734 raft_consensus.cc:2379] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.690006 1737 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.699703 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1161) took 1070 ms. Trace:
I20260927 23:31:02.729554 1579 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.729915 1582 rpcz_store.cc:276] 0927 23:31:01.629108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.355797 (+726689us) service_pool.cc:227] Handling call
0927 23:31:02.356121 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.356414 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:02.356439 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:02.356939 (+ 500us) log.cc:842] Serialized 26 byte log entry
0927 23:31:02.498811 (+141872us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:02.544719 (+ 45908us) log.cc:842] Serialized 14 byte log entry
0927 23:31:02.546457 (+ 1738us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:02.546634 (+ 177us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:02.694077 (+147443us) raft_consensus.cc:1671] finished
0927 23:31:02.694096 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:02.699641 (+ 5545us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1507,"prepare.queue_time_us":102588,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":7735,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":42456,"thread_start_us":60479,"threads_started":2,"wal-append.queue_time_us":190217}
I20260927 23:31:02.732482 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.661062 3074 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.693821 1728 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:02.665256 3146 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:02.691982 3270 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260927 23:31:02.671518 3193 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:02.742698 3193 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.743337 3243 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.744653 3193 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.671492 3015 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.744928 3224 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.745441 3247 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.672104 3104 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.613449 3108 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.706728 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.707603 3097 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.714847 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.722694 3137 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.605306 1583 rpcz_store.cc:276] 0927 23:31:01.375041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.379871 (+ 4830us) service_pool.cc:227] Handling call
0927 23:31:01.458136 (+ 78265us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:01.495674 (+ 37538us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:01.495707 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:01.496229 (+ 522us) log.cc:842] Serialized 26 byte log entry
0927 23:31:02.023587 (+527358us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:02.034306 (+ 10719us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:02.046214 (+ 11908us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:02.286299 (+240085us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:02.516149 (+229850us) raft_consensus.cc:1671] finished
0927 23:31:02.516180 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:02.542959 (+ 26779us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10926,"raft.queue_time_us":20090,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"spinlock_wait_cycles":19866624,"thread_start_us":490623,"threads_started":1,"wal-append.queue_time_us":511131}
I20260927 23:31:02.620306 3135 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.764542 3135 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.764942 3135 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.660858 3091 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.770843 3040 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.770991 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:02.771664 2289 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.771791 3091 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.665648 3207 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.773895 3207 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.711122 3256 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.705188 2905 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.749100 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.778220 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.778851 1737 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:02.750916 3072 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:02.751483 2287 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 303) took 1636 ms (client timeout 2072 ms). Trace:
I20260927 23:31:02.757437 3276 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.753661 1584 raft_consensus.cc:2354] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:02.757624 3125 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:02.723627 3253 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.705219 3075 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.782778 1733 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.735244 3153 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.747977 3146 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:02.791533 2910 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.758580 3104 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.761287 1580 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.795060 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:02.743232 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:02.737524 3114 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.782109 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:02.797153 3247 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.797356 3214 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.798262 3108 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.787931 3153 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:02.781903 2287 rpcz_store.cc:269] 0927 23:31:01.115485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.133775 (+ 18290us) service_pool.cc:227] Handling call
0927 23:31:02.751299 (+1617524us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54004e0cd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:02.751407 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2680412544}
I20260927 23:31:02.791368 3135 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.758081 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.836795 3135 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.797261 1737 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.840492 1731 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:02.796849 3276 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:02.784520 2258 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.795686 1580 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:02.842967 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:02.843305 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 58866.9us
I20260927 23:31:02.843570 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:02.843938 1583 raft_consensus.cc:2396] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260927 23:31:02.844295 1580 raft_consensus.cc:2379] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 2: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:02.844626 1583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:02.839226 3153 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.845155 1583 raft_consensus.cc:595] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:02.845608 1583 raft_consensus.cc:603] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:02.846375 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.848615 3247 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:02.849045 3247 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:02.849455 3247 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:02.853259 3052 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.528s user 0.006s sys 0.003s
I20260927 23:31:02.853540 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:02.857549 3176 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.532s user 0.009s sys 0.000s
I20260927 23:31:02.868870 1587 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:02.784584 2288 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:02.878034 3153 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.808425 3171 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.927116 1658 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:02.926008 3153 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.929028 3171 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.879755 3174 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:02.928294 3114 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.939815 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:02.941823 2287 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:02.943454 1629 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:02.944556 3171 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:02.945040 3171 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:02.945539 3171 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:02.933614 1728 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:02.954248 3276 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:02.954865 3276 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:02.957090 3276 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:02.959347 3224 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:02.963460 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:02.963766 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:02.899431 1587 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:03.007418 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:03.008950 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45250.7us
I20260927 23:31:03.013557 1737 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:03.009439 1587 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:03.014974 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.015481 2259 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:03.016141 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.017753 2258 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.018309 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.013597 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:02.853778 3052 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.020963 1662 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:03.016894 1661 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:31:03.031351 3145 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:03.007997 3214 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.031988 3145 raft_consensus.cc:686] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:03.032138 3214 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.004240 3176 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:03.035573 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.034538 3096 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.008622 3224 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:03.006500 3153 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.019829 3262 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:03.033015 3145 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.005327 3098 raft_consensus.cc:686] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:03.010901 2910 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:03.034121 3214 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.026747 3243 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:03.084597 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.049294 3096 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.096258 3277 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:03.100754 3134 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.102941 2291 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260927 23:31:03.102702 3098 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.096613 2910 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.105383 3177 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:03.105234 3171 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.105902 1654 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:03.091084 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.107950 2291 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:31:03.108122 1660 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:03.011466 3204 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.117378 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.089020 3224 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.090198 3262 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.029230 3135 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20260927 23:31:03.106539 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 175) took 1151 ms. Trace:
I20260927 23:31:03.123960 1552 leader_election.cc:398] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:03.107682 3096 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.105893 3177 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:03.052476 3176 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.137095 1552 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.111006 2287 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:03.132637 3262 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.102705 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.094182 3146 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:03.122514 1659 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.001s sys 0.007s
I20260927 23:31:03.153587 1552 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.132408 3214 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:03.142752 1580 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:03.155864 3281 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.156597 3052 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.158473 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1299) took 1075 ms. Trace:
I20260927 23:31:03.159549 3176 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.160116 3174 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.162880 1659 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.001s sys 0.007s
I20260927 23:31:03.107851 3104 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.107025 3243 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.163447 3134 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.167335 1703 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.128825 3153 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260927 23:31:03.170497 1552 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:03.168069 3281 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.171799 3282 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.166682 1579 rpcz_store.cc:276] 0927 23:31:02.082880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.214232 (+131352us) service_pool.cc:227] Handling call
0927 23:31:03.158306 (+944074us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b54003318d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.158421 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":948814080}
I20260927 23:31:03.175895 3125 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:03.175901 1732 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:03.176893 3224 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.129608 3276 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260927 23:31:03.177624 3104 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.178414 1582 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:31:03.049966 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 401872us
I20260927 23:31:03.186061 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.171188 3214 raft_consensus.cc:686] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:03.166275 2910 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.179399 3128 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:03.200389 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:03.202003 3125 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.197178 3282 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.233469 3282 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.233629 3214 consensus_queue.cc:237] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.235756 3128 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.237679 3253 raft_consensus.cc:3032] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:03.238077 3253 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:03.238920 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1467) took 1027 ms. Trace:
I20260927 23:31:03.239004 3253 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.239449 1658 rpcz_store.cc:276] 0927 23:31:02.211534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.252648 (+ 41114us) service_pool.cc:227] Handling call
0927 23:31:03.238742 (+986094us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b5400447fd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.238862 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1549074304}
I20260927 23:31:03.240366 3253 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.242756 3243 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.244638 3146 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.246138 3114 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.246603 3114 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:03.246819 3199 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.605s user 0.008s sys 0.001s
I20260927 23:31:03.247215 3199 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.202348 3171 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.180975 3177 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.190997 3281 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.203743 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:03.237999 3128 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.249370 3199 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.198575 3109 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.007s sys 0.038s
I20260927 23:31:03.277379 1627 leader_election.cc:398] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:03.185348 3072 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:03.211616 1702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.007s sys 0.037s
I20260927 23:31:03.189518 2905 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.135951 1583 rpcz_store.cc:276] 0927 23:31:01.955071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.846242 (+891171us) service_pool.cc:227] Handling call
0927 23:31:03.030136 (+183894us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:03.052782 (+ 22646us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:03.086631 (+ 33849us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:03.091549 (+ 4918us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:03.094085 (+ 2536us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.096806 (+ 2721us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:03.106484 (+ 9678us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:03.189940 3135 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:03.231845 1579 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
W20260927 23:31:03.260046 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 74080.3us
I20260927 23:31:03.164788 3143 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.275697 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.277743 1627 leader_election.cc:401] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:03.278168 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.279986 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:03.283363 1702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.007s sys 0.044s
I20260927 23:31:03.263280 3146 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.268247 2287 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.363869 1700 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:03.364142 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 86850.3us
I20260927 23:31:03.372807 1627 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:03.276762 3184 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.671s user 0.009s sys 0.001s
I20260927 23:31:03.369249 1703 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.364674 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.284119 3153 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260927 23:31:03.336917 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.375090 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.375313 1579 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.375790 1735 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:03.376497 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.376649 3153 consensus_queue.cc:1046] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:03.377506 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.378021 1735 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.267182 3177 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.379086 3118 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.379591 3118 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.381610 3118 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.360204 3258 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260927 23:31:03.390800 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:03.339865 3109 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s user 0.007s sys 0.040s
I20260927 23:31:03.360307 3146 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:03.371151 1730 consensus_meta.cc:287] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.748s user 0.006s sys 0.006s
I20260927 23:31:03.394374 1733 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:03.372366 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.359535 3135 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.270730 3171 consensus_queue.cc:1046] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.113s
I20260927 23:31:03.349675 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.370836 3176 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:03.480916 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.335399 1662 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:03.488143 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.488866 1662 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.335584 1661 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:03.472625 1579 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:03.495164 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.496208 1661 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 1.
I20260927 23:31:03.496744 2259 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:03.497180 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:03.478943 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 810372us
I20260927 23:31:03.495314 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.497974 1661 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.498211 1579 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:03.475988 3289 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
W20260927 23:31:03.395025 2289 consensus_meta.cc:287] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.623s user 0.008s sys 0.002s
I20260927 23:31:03.398825 3108 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:03.504833 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.412189 3146 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:03.392454 3247 consensus_meta.cc:287] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.543s user 0.011s sys 0.000s
I20260927 23:31:03.391281 3283 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:03.339589 3072 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.533303 3176 raft_consensus.cc:3032] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:03.533579 3288 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.257s user 0.000s sys 0.000s
I20260927 23:31:03.399592 3204 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.537351 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.477134 3135 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.541589 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.542402 1661 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:03.540426 1662 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:03.542280 1579 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:03.483498 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:03.472640 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.505987 3052 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.525209 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:03.574839 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 200196us
I20260927 23:31:03.530103 3277 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:03.544478 3174 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.478847 1730 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:03.565567 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.387297 3184 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.564985 1733 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:03.540328 2289 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:03.583688 1656 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.584605 1627 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:03.587128 2294 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:03.581489 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.587800 1579 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.543545 3054 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.580019 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:03.583642 1662 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:03.542848 3108 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.589494 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1447) took 1106 ms. Trace:
I20260927 23:31:03.591727 3108 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.591961 1730 rpcz_store.cc:276] 0927 23:31:02.483279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.617925 (+134646us) service_pool.cc:227] Handling call
0927 23:31:03.586704 (+968779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:03.592849 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.593272 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.596364 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1170) took 1257 ms. Trace:
I20260927 23:31:03.567720 1728 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:03.596885 1734 rpcz_store.cc:276] 0927 23:31:02.338645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.737513 (+398868us) service_pool.cc:227] Handling call
0927 23:31:02.737911 (+ 398us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.749790 (+ 11879us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:02.749832 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:02.758428 (+ 8596us) log.cc:842] Serialized 26 byte log entry
0927 23:31:03.186648 (+428220us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:03.186700 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.186785 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:03.596141 (+409356us) raft_consensus.cc:1671] finished
0927 23:31:03.596161 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:03.596296 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11439,"raft.queue_time_us":12001,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":402610,"threads_started":1,"wal-append.queue_time_us":762252}
I20260927 23:31:03.593971 1730 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:03.578496 3125 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.614459 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:03.564836 3176 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:03.617252 1552 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.566912 1583 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:03.610185 1657 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.611598 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.611384 1703 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.612890 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.621615 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.622395 1734 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:03.592067 3292 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260927 23:31:03.634131 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:03.639875 1579 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:03.568575 3096 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.545962 3247 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.594269 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:03.604930 3054 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.605808 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.654495 1656 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:03.628489 3277 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:03.586915 3174 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.635778 3262 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.691957 1661 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:03.698083 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1450) took 1047 ms. Trace:
I20260927 23:31:03.676734 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.671793 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.649704 3128 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:03.647033 3219 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:03.657611 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.659242 1657 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:03.645458 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:03.629662 3214 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:03.655184 3199 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:03.761291 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1163) took 1823 ms. Trace:
I20260927 23:31:03.762691 1656 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260927 23:31:03.763046 1656 pending_rounds.cc:76] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 4
I20260927 23:31:03.762974 1580 rpcz_store.cc:276] 0927 23:31:01.938049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.844779 (+906730us) service_pool.cc:227] Handling call
0927 23:31:02.845053 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:03.169091 (+324038us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.760989 (+591898us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b540038e0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:03.761218 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684213632}
W20260927 23:31:03.746122 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 266825us
I20260927 23:31:03.655710 1734 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:03.727574 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1365) took 1394 ms. Trace:
I20260927 23:31:03.765126 1663 rpcz_store.cc:276] 0927 23:31:02.332819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.472455 (+139636us) service_pool.cc:227] Handling call
0927 23:31:03.727355 (+1254900us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54002dcdd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.727501 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152183680}
I20260927 23:31:03.765380 1628 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:03.766017 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.633478 3283 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:03.749285 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 77583.1us
I20260927 23:31:03.698549 3075 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.762436 3184 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.649260 3125 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:03.709220 1658 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:03.743865 1737 rpcz_store.cc:276] 0927 23:31:02.650568 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.014544 (+363976us) service_pool.cc:227] Handling call
0927 23:31:03.014844 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:03.015188 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:03.015294 (+ 106us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:03.027477 (+ 12183us) log.cc:842] Serialized 26 byte log entry
0927 23:31:03.221276 (+193799us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:03.221332 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.221432 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:03.697872 (+476440us) raft_consensus.cc:1671] finished
0927 23:31:03.697892 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:03.698027 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41661,"raft.queue_time_us":4330,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":10046,"thread_start_us":135894,"threads_started":1,"wal-append.queue_time_us":345981}
I20260927 23:31:03.755374 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1173) took 1415 ms. Trace:
I20260927 23:31:03.757397 3108 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.760459 3128 raft_consensus.cc:686] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:03.725092 3260 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.247s user 0.000s sys 0.013s
I20260927 23:31:03.745678 1733 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.761528 2259 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:03.767182 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.673367 2290 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.720654 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.647907 3143 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.712209 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 208906us
I20260927 23:31:03.770766 3199 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.695081 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.863385 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.863963 1730 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.680585 2910 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:03.707495 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:03.868146 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.868954 1583 raft_consensus.cc:2354] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:03.869771 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:03.763950 1656 pending_rounds.cc:97] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260927 23:31:03.870489 1656 raft_consensus.cc:2866] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:03.771083 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:03.746279 3072 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.776446 1735 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:03.872748 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.873395 1735 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:03.696698 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:03.762127 3293 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260927 23:31:03.882786 1580 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:03.753300 3224 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:03.781453 3117 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.759704 3214 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:03.728611 3281 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.760385 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:03.666203 3054 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.797202 3174 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:03.792217 3298 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:03.806308 3164 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:03.916811 3164 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:03.808984 3096 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:03.694507 3176 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.811722 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:03.813056 3262 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.812810 3146 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.785271 1731 rpcz_store.cc:276] 0927 23:31:02.339566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.541534 (+201968us) service_pool.cc:227] Handling call
0927 23:31:03.755153 (+1213619us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400104d54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.755300 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514895104}
I20260927 23:31:03.787411 3283 raft_consensus.cc:686] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:03.980613 3283 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:03.981829 3284 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.507s user 0.001s sys 0.001s
I20260927 23:31:03.815201 3108 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:03.816061 3114 consensus_meta.cc:287] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.569s user 0.005s sys 0.005s
I20260927 23:31:03.982676 3114 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.984254 2289 raft_consensus.cc:2354] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:03.984987 3114 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.815843 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 1.22128e+06us
I20260927 23:31:03.985908 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.818014 2901 consensus_queue.cc:1046] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260927 23:31:03.818923 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1173) took 1436 ms (client timeout 1606 ms). Trace:
I20260927 23:31:03.759347 3219 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:03.783845 3184 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:31:03.823333 3297 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.257s user 0.002s sys 0.000s
I20260927 23:31:03.823577 1733 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:04.011579 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 246270us
I20260927 23:31:04.012095 1554 leader_election.cc:398] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.011916 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.012432 1554 leader_election.cc:401] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:03.781486 3171 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:03.788311 1732 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
W20260927 23:31:03.790165 3282 consensus_meta.cc:287] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.556s user 0.001s sys 0.010s
I20260927 23:31:03.853219 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1154) took 2624 ms. Trace:
I20260927 23:31:04.024845 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.768607 1734 raft_consensus.cc:2354] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.026722 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:04.020346 3310 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
W20260927 23:31:03.857249 3267 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.242s user 0.001s sys 0.011s
I20260927 23:31:03.855473 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:04.018946 3108 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:03.881475 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:04.037276 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.020771 2289 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:03.883625 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:03.886068 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 115057us
I20260927 23:31:03.914863 1579 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:31:04.035164 2905 consensus_meta.cc:287] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.661s user 0.010s sys 0.001s
I20260927 23:31:03.785733 3125 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:03.917943 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 333452us
I20260927 23:31:04.039673 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:03.937857 3128 consensus_queue.cc:237] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.930533 3214 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.880796 3302 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.918210 2910 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:03.977566 3262 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:03.978366 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1167) took 1881 ms (client timeout 1673 ms). Trace:
W20260927 23:31:03.979640 3306 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260927 23:31:03.986683 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:03.989264 3176 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:03.816416 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.989992 3126 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260927 23:31:03.995569 3146 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:03.997148 3199 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:03.999605 1654 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1468) took 1758 ms (client timeout 1693 ms). Trace:
I20260927 23:31:04.000979 3052 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:04.002864 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:04.007668 1581 rpcz_store.cc:269] 0927 23:31:02.385972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.477741 (+ 91769us) service_pool.cc:227] Handling call
0927 23:31:03.818606 (+1340865us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b540009ecd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.818743 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327069568}
I20260927 23:31:04.039645 3108 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.890146 1728 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:03.885206 3224 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:03.884271 1583 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:03.883081 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:03.856678 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1162) took 1523 ms. Trace:
I20260927 23:31:04.030155 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.024766 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1361) took 1693 ms. Trace:
I20260927 23:31:03.917631 3164 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.027007 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:04.022254 1588 rpcz_store.cc:276] 0927 23:31:01.200145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:01.200614 (+ 469us) service_pool.cc:227] Handling call
0927 23:31:01.200997 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:02.597473 (+1396476us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.824355 (+1226882us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540030c854. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:03.824591 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3103216000}
W20260927 23:31:04.035392 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 180012us
I20260927 23:31:04.016727 3171 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:03.878876 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1157) took 2253 ms. Trace:
I20260927 23:31:03.820932 3247 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.056025 3307 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.043332 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:04.219107 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 175835us
I20260927 23:31:04.218952 3307 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.219739 1552 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:04.220163 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:04.220378 3146 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.220815 1552 leader_election.cc:398] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.220880 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:03.916160 3281 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.221022 3061 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
W20260927 23:31:04.221536 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1302) took 1834 ms (client timeout 2317 ms). Trace:
I20260927 23:31:03.930048 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:04.044253 1628 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:04.222164 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:04.222437 2294 raft_consensus.cc:2379] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:04.031381 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.222738 1737 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.222995 1731 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:04.039374 3117 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:04.223775 3194 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.223877 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.224346 3194 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.048611 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1165) took 1712 ms. Trace:
I20260927 23:31:04.211655 3098 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:04.051012 1735 raft_consensus.cc:2379] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:04.036031 3206 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:04.054193 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:04.087110 2287 consensus_meta.cc:287] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.690s user 0.009s sys 0.001s
I20260927 23:31:04.025307 3282 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:03.889460 3277 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.055974 2292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 314) took 1318 ms. Trace:
I20260927 23:31:04.228807 1729 rpcz_store.cc:276] 0927 23:31:02.335878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.336020 (+ 142us) service_pool.cc:227] Handling call
0927 23:31:02.336388 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.336639 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:02.336674 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:02.364567 (+ 27893us) log.cc:842] Serialized 26 byte log entry
0927 23:31:02.530124 (+165557us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:02.530158 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:02.530224 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:03.856680 (+1326456us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:04.046111 (+189431us) raft_consensus.cc:1671] finished
0927 23:31:04.046132 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.048544 (+ 2412us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13116288,"thread_start_us":114073,"threads_started":1,"wal-append.queue_time_us":181046}
W20260927 23:31:04.237052 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 182938us
I20260927 23:31:04.056111 3219 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.240599 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 313) took 1743 ms. Trace:
I20260927 23:31:04.220760 1732 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.106002 1663 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:04.243566 1735 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:04.178874 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:04.179255 1656 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:04.092360 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:04.159322 1551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.693s (SENT) [suppressed 52 similar messages]
I20260927 23:31:04.244604 1581 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:04.054929 3253 consensus_meta.cc:287] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.814s user 0.013s sys 0.000s
I20260927 23:31:04.241134 2293 rpcz_store.cc:276] 0927 23:31:02.497142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.497330 (+ 188us) service_pool.cc:227] Handling call
0927 23:31:02.497723 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:02.687093 (+189370us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.240355 (+1553262us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400540454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:04.240527 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334416768}
I20260927 23:31:04.096051 3125 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.180655 1587 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:04.055642 2910 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:04.200488 3052 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.041388 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:04.315802 1579 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.318652 1580 raft_consensus.cc:2379] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:04.052248 3315 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:04.214730 3171 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
W20260927 23:31:04.101126 2285 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 311) took 1842 ms (client timeout 1965 ms). Trace:
I20260927 23:31:04.217031 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.217586 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:04.216089 1702 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:04.218051 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 131) took 1205 ms. Trace:
I20260927 23:31:04.177845 1736 rpcz_store.cc:276] 0927 23:31:02.333630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.333751 (+ 121us) service_pool.cc:227] Handling call
0927 23:31:02.470583 (+136832us) spinlock_profiling.cc:243] Waited 984 us on lock 0x7b5800282098. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:03.856485 (+1385902us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b54002bd254. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.856625 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112574464}
I20260927 23:31:04.221275 1552 leader_election.cc:401] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260927 23:31:04.220883 3146 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:04.222594 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.045135 2905 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:04.042616 3177 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260927 23:31:04.101235 3143 raft_consensus.cc:3032] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:04.036151 3072 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 4
W20260927 23:31:04.222744 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 178559us
W20260927 23:31:04.084357 1654 rpcz_store.cc:269] 0927 23:31:02.241640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.248511 (+ 6871us) service_pool.cc:227] Handling call
0927 23:31:03.999373 (+1750862us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b54000500d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.999531 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1666564480}
I20260927 23:31:04.225883 3239 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:04.225097 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:04.178067 1659 rpcz_store.cc:276] 0927 23:31:02.331696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.551252 (+219556us) service_pool.cc:227] Handling call
0927 23:31:02.551530 (+ 278us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.551829 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:02.551856 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:02.572907 (+ 21051us) log.cc:842] Serialized 26 byte log entry
0927 23:31:04.005290 (+1432383us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b1800375e20. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f6411
0927 23:31:04.005504 (+ 214us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:04.005565 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.005661 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:04.005708 (+ 47us) raft_consensus.cc:1671] finished
0927 23:31:04.005722 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.024665 (+ 18943us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7562,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4020,"spinlock_wait_cycles":8792064,"thread_start_us":601686,"threads_started":1,"wal-append.queue_time_us":601187}
I20260927 23:31:04.104671 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 132) took 1091 ms. Trace:
I20260927 23:31:04.245013 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.181697 3224 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:04.247253 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.096213 3311 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.421901 1657 rpcz_store.cc:276] 0927 23:31:03.013143 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.014717 (+ 1574us) service_pool.cc:227] Handling call
0927 23:31:04.104457 (+1089740us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b540013b5d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:04.104609 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1367025024}
I20260927 23:31:04.103053 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:04.102734 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.236202 2292 rpcz_store.cc:276] 0927 23:31:02.738498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.738650 (+ 152us) service_pool.cc:227] Handling call
0927 23:31:02.738999 (+ 349us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.739361 (+ 362us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:02.739398 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:02.752435 (+ 13037us) log.cc:842] Serialized 26 byte log entry
0927 23:31:03.241989 (+489554us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18002b40e0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db29345 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f6411 00007f9750a435cf
0927 23:31:03.397336 (+155347us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:03.398806 (+ 1470us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.398874 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:03.822811 (+423937us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:04.055758 (+232947us) raft_consensus.cc:1671] finished
0927 23:31:04.055774 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.055908 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":679,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":5637,"spinlock_wait_cycles":32958848,"thread_start_us":155173,"threads_started":1,"wal-append.queue_time_us":806424}
I20260927 23:31:04.210969 1585 rpcz_store.cc:276] 0927 23:31:01.625728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.319358 (+693630us) service_pool.cc:227] Handling call
0927 23:31:02.319601 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:02.489442 (+169841us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:03.878580 (+1389138us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400466754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:03.878798 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320662272}
I20260927 23:31:04.312064 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 181) took 1345 ms. Trace:
I20260927 23:31:04.427158 1587 rpcz_store.cc:276] 0927 23:31:02.966377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.014885 (+ 48508us) service_pool.cc:227] Handling call
0927 23:31:04.002675 (+987790us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b5800001898. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:04.312003 (+309328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14887680}
I20260927 23:31:04.428007 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:04.428390 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:04.428819 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 183) took 1461 ms. Trace:
I20260927 23:31:04.429174 1585 rpcz_store.cc:276] 0927 23:31:02.967081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.427779 (+1460698us) service_pool.cc:227] Handling call
0927 23:31:04.428766 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.054509 3302 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.360780 2910 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:04.431989 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 387266us
I20260927 23:31:04.429893 1585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
W20260927 23:31:04.432255 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 272965us
I20260927 23:31:04.253511 1734 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:04.432710 1585 raft_consensus.cc:595] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.433758 1734 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:04.433773 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:04.424021 1656 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:04.435262 1734 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 23:31:04.315038 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 66710.8us
I20260927 23:31:04.237093 2287 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:04.053965 1584 rpcz_store.cc:269] 0927 23:31:02.097119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.122025 (+ 24906us) service_pool.cc:227] Handling call
0927 23:31:03.978064 (+1856039us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b5400330ed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:03.978264 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":965900672}
I20260927 23:31:04.327564 3307 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.312000 3219 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:04.328315 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1160) took 2699 ms. Trace:
I20260927 23:31:04.222443 3061 raft_consensus.cc:686] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:04.063580 3314 consensus_queue.cc:1046] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.197s
I20260927 23:31:04.075379 3262 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.313355 1658 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:04.339501 1661 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.487120 1736 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:04.347234 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:04.343271 1733 raft_consensus.cc:2354] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.436993 3194 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.340533 1660 rpcz_store.cc:276] 0927 23:31:03.012848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.019237 (+ 6389us) service_pool.cc:227] Handling call
0927 23:31:04.217887 (+1198650us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b5400446bd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:04.218006 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138251520}
W20260927 23:31:04.346488 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 125778us
I20260927 23:31:04.496338 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.087173 3135 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.341780 1579 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:04.499351 1733 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:04.500165 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1172) took 2118 ms. Trace:
I20260927 23:31:04.500490 1579 rpcz_store.cc:276] 0927 23:31:02.382189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.499531 (+2117342us) service_pool.cc:227] Handling call
0927 23:31:04.499761 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.499893 (+ 132us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.499917 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.499953 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:04.499980 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.500026 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.500107 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.501892 1579 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:04.246883 3117 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:04.353914 1581 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:04.354945 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1373) took 1071 ms. Trace:
I20260927 23:31:04.502610 1662 rpcz_store.cc:276] 0927 23:31:03.283853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.590734 (+306881us) service_pool.cc:227] Handling call
0927 23:31:03.979955 (+389221us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:03.980205 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.010168 (+ 29963us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.010197 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.010240 (+ 43us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:04.010268 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.354716 (+344448us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b5400447ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:04.354773 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.354896 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102681216}
I20260927 23:31:04.503851 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:04.354897 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 138888us
I20260927 23:31:04.473088 3247 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.249336 3199 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.356918 3283 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:04.362942 1628 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:04.221972 1582 rpcz_store.cc:269] 0927 23:31:02.387128 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.732185 (+345057us) service_pool.cc:227] Handling call
0927 23:31:04.221377 (+1489192us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b540007ce54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:04.221486 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1023325440}
I20260927 23:31:04.517225 1582 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:04.517807 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1181) took 1121 ms. Trace:
I20260927 23:31:04.518093 1582 rpcz_store.cc:276] 0927 23:31:03.396802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.516817 (+1120015us) service_pool.cc:227] Handling call
0927 23:31:04.517068 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.517600 (+ 532us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.517760 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.489809 3301 raft_consensus.cc:3032] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:04.247186 1730 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:31:04.518956 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 156096us
I20260927 23:31:04.421299 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.338667 3164 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.048424 3054 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.438694 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 12) took 1249 ms. Trace:
I20260927 23:31:04.436429 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.521159 1585 rpcz_store.cc:276] 0927 23:31:03.184847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.429778 (+1244931us) service_pool.cc:227] Handling call
0927 23:31:04.433842 (+ 4064us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.429787 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:04.517225 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.522125 1660 raft_consensus.cc:2354] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.522204 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1183) took 1045 ms. Trace:
I20260927 23:31:04.522703 1587 rpcz_store.cc:276] 0927 23:31:03.476586 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.429524 (+952938us) service_pool.cc:227] Handling call
0927 23:31:04.522141 (+ 92617us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.436393 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.525260 1734 raft_consensus.cc:1261] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:04.525822 1585 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:04.443142 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.426553 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.467005 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.527119 1657 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.467518 1551 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:04.528306 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 60882.1us
I20260927 23:31:04.529421 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:04.338886 2285 rpcz_store.cc:269] 0927 23:31:02.258450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.288700 (+ 30250us) service_pool.cc:227] Handling call
0927 23:31:04.100920 (+1812220us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54006437d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:04.101056 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251121024}
I20260927 23:31:04.472806 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 178) took 2265 ms. Trace:
I20260927 23:31:04.432037 3302 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.475781 1586 rpcz_store.cc:276] 0927 23:31:01.628780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:02.320963 (+692183us) service_pool.cc:227] Handling call
0927 23:31:02.321235 (+ 272us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:02.693919 (+372684us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.328065 (+1634146us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400241b54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:04.328258 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":855820672}
I20260927 23:31:04.478363 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1455) took 1112 ms. Trace:
I20260927 23:31:04.238615 3206 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:04.339540 1737 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.355638 3143 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:04.491954 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 51626.5us
I20260927 23:31:04.437783 3145 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.436771 3313 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.515538 3194 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260927 23:31:04.466879 2906 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.317963 3281 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:04.475597 2287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 318) took 1270 ms. Trace:
I20260927 23:31:04.510123 1736 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:04.495759 1655 raft_consensus.cc:2379] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:04.476095 3279 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:04.472463 3118 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:04.509544 1661 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.518762 3247 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.585618 3307 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:04.591866 3307 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.593930 3307 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.595369 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.583192 3239 raft_consensus.cc:686] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:04.597278 3224 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.597378 3206 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.597688 3239 consensus_queue.cc:237] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.593716 3171 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.349607 3312 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.158s
I20260927 23:31:04.520068 1659 raft_consensus.cc:2396] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:04.520191 1628 leader_election.cc:398] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:04.354380 3125 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.521351 1656 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.438774 2259 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:04.531340 1582 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:04.338495 3315 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.532050 2292 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:04.532910 1580 rpcz_store.cc:276] 0927 23:31:02.207760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.467469 (+2259709us) service_pool.cc:227] Handling call
0927 23:31:04.472368 (+ 4899us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.472531 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.472554 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.472593 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:04.472622 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.472671 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.472759 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.433933 3311 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.661228 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.661854 1659 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 1.
I20260927 23:31:04.662973 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.533929 1657 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.667580 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.668565 1657 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:04.478058 3134 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260927 23:31:04.670894 3134 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:04.536149 1735 rpcz_store.cc:276] 0927 23:31:03.366041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.233946 (+867905us) service_pool.cc:227] Handling call
0927 23:31:04.234285 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.244188 (+ 9903us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.478138 (+233950us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b5400104854. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:04.478290 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20852608}
I20260927 23:31:04.513877 3175 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.672201 3175 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.516350 3283 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:04.488750 3219 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.540445 1583 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:04.530244 3108 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:04.530464 3252 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:04.533797 3314 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.525399 3301 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:04.488885 3146 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.239111 3098 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.690972 3098 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.691152 3315 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.557379 3313 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.691731 3315 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:31:04.692106 3135 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.692309 3315 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:04.692691 3315 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.693039 3315 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.693312 3313 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.645370 3224 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.657719 3145 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:04.560168 3184 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.544s user 0.009s sys 0.002s
I20260927 23:31:04.695832 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.696147 3145 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.561779 3072 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:04.562686 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1171) took 2180 ms. Trace:
I20260927 23:31:04.534077 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1187) took 1171 ms. Trace:
I20260927 23:31:04.514544 3117 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:04.552270 2290 consensus_meta.cc:287] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.670s user 0.008s sys 0.002s
I20260927 23:31:04.584717 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.760617 1661 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:04.729811 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 233547us
I20260927 23:31:04.515195 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1174) took 2119 ms. Trace:
I20260927 23:31:04.539479 3199 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:04.646137 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 207460us
I20260927 23:31:04.766975 2259 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.589736 3279 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.647399 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 187) took 1100 ms. Trace:
I20260927 23:31:04.771494 1582 rpcz_store.cc:276] 0927 23:31:03.546875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.518814 (+971939us) service_pool.cc:227] Handling call
0927 23:31:04.522917 (+ 4103us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.647062 (+124145us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.647330 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.658521 1581 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 5. Ops: [3.6-3.6]
I20260927 23:31:04.531778 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.687469 1659 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260927 23:31:04.775548 1659 pending_rounds.cc:76] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 1
I20260927 23:31:04.775847 1659 pending_rounds.cc:97] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260927 23:31:04.696192 3184 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:31:04.688669 3321 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260927 23:31:04.688349 3301 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.689007 3171 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.691421 3262 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.694834 3052 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.544673 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:04.685671 1656 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:04.587360 3054 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.717783 1657 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 23:31:04.587945 2901 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:04.586246 3282 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.578423 2906 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.578174 2287 rpcz_store.cc:276] 0927 23:31:03.203338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.203496 (+ 158us) service_pool.cc:227] Handling call
0927 23:31:04.473790 (+1270294us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:04.757208 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 227836us
I20260927 23:31:04.756227 1730 rpcz_store.cc:276] 0927 23:31:03.362772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.863117 (+500345us) service_pool.cc:227] Handling call
0927 23:31:04.531686 (+668569us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.762928 1552 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.763373 1579 rpcz_store.cc:276] 0927 23:31:02.386164 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.501298 (+2115134us) service_pool.cc:227] Handling call
0927 23:31:04.501634 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:04.502290 (+ 656us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.505571 (+ 3281us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:04.694169 3316 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.474s user 0.001s sys 0.002s
I20260927 23:31:04.677152 3219 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:04.768395 2259 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:04.646703 1628 leader_election.cc:401] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260927 23:31:04.837181 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 317051us
I20260927 23:31:04.837946 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.839090 3312 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
W20260927 23:31:04.839737 3214 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.793s user 0.007s sys 0.003s
I20260927 23:31:04.840181 3214 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.839740 3312 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:04.841396 3312 raft_consensus.cc:686] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:04.674818 3283 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.842248 3312 consensus_queue.cc:237] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.844172 3214 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.675927 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1305) took 1560 ms. Trace:
I20260927 23:31:04.676779 3314 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:04.674310 3175 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:04.776139 1659 raft_consensus.cc:2866] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:04.548048 3061 consensus_queue.cc:237] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.777849 1661 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.872027 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.736544 3098 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:04.782301 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 213154us
I20260927 23:31:04.873661 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:04.785235 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 158303us
W20260927 23:31:04.868535 1628 connection.cc:232] RPC call timeout handler was delayed by -1.2407546450000382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
I20260927 23:31:04.876475 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:04.877010 1628 connection.cc:232] RPC call timeout handler was delayed by -1.0727446070000042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
I20260927 23:31:04.770933 1655 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.765408 2290 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:04.589608 3118 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.799841 1656 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:04.804597 1551 leader_election.cc:398] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
W20260927 23:31:04.884059 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 79590.9us
I20260927 23:31:04.884239 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.884985 1656 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.886965 1551 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:04.887555 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.720320 1588 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:04.676417 3252 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:04.804837 1662 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:04.820206 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.851109 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:04.823098 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 60322.1us
I20260927 23:31:04.800882 2288 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.824566 3117 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:04.825006 3279 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.854890 2289 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:04.870951 3307 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.830269 1654 raft_consensus.cc:1261] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 23:31:04.831580 1728 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.603s user 0.006s sys 0.002s
I20260927 23:31:04.860237 1583 rpcz_store.cc:276] 0927 23:31:03.115759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.235290 (+1119531us) service_pool.cc:227] Handling call
0927 23:31:04.675834 (+440544us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.929244 1486 catalog_manager.cc:5495] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:04.852393 3315 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:04.833387 3176 consensus_meta.cc:287] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.756s user 0.003s sys 0.005s
I20260927 23:31:04.833490 2906 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.834939 3052 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:04.802938 3054 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.835618 3247 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.770711 1585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:04.716886 3143 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.853490 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.953066 1660 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 3
I20260927 23:31:04.851843 1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48998 (request call id 231) took 9109 ms. Trace:
W20260927 23:31:04.860359 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 92041.1us
I20260927 23:31:04.849753 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.851841 2292 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 23:31:04.759287 1584 rpcz_store.cc:276] 0927 23:31:02.381968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.492979 (+2111011us) service_pool.cc:227] Handling call
0927 23:31:04.544771 (+ 51792us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.554384 (+ 9613us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.562351 (+ 7967us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.562411 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:04.562454 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.562520 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:04.562632 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.857174 1629 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:04.960075 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 103128us
I20260927 23:31:04.872963 1661 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:04.873826 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.878062 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:04.847250 2901 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:04.962566 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 84557.2us
I20260927 23:31:04.946236 3239 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.844762 3335 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:04.894639 1656 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:04.971115 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.950690 3274 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:04.975924 2292 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:04.975766 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.977016 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.970059 3052 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.977694 1656 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:04.970543 3247 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.978300 3052 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:04.978518 3054 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.970656 3143 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.978746 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.846890 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:04.979256 3052 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:04.981176 1656 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:04.898414 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.982234 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.898651 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:04.826427 3301 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:04.826778 3206 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.852598 3280 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260927 23:31:04.831944 3199 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.946949 1486 catalog_manager.cc:5495] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:04.899492 3214 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:04.900746 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.951663 1585 raft_consensus.cc:595] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:04.901232 2290 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51538 (request call id 379) took 1498 ms (client timeout 1719 ms). Trace:
I20260927 23:31:04.889863 1551 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:04.963109 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:05.039759 2294 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:04.963837 1700 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:05.040020 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 150235us
W20260927 23:31:05.040339 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 76600.7us
I20260927 23:31:05.041313 1700 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:04.862751 3262 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:04.947466 3343 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:04.966251 1660 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:05.053045 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1483) took 1030 ms. Trace:
W20260927 23:31:04.968228 3329 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20260927 23:31:04.938103 1728 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.936268 2289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 322) took 1137 ms. Trace:
I20260927 23:31:04.969010 3279 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
W20260927 23:31:04.982560 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 84292.7us
W20260927 23:31:04.880427 1732 consensus_meta.cc:287] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.555s user 0.007s sys 0.003s
I20260927 23:31:05.074625 1732 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:04.983145 1656 raft_consensus.cc:2379] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:05.075368 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1477) took 1032 ms. Trace:
I20260927 23:31:05.075487 3175 raft_consensus.cc:2741] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:31:05.076035 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.983085 1659 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.076745 1656 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:05.077625 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:04.936381 3307 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.078890 3252 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.079013 3108 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:05.065866 3338 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260927 23:31:05.079736 3307 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.079574 3108 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.054426 3163 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260927 23:31:05.080446 3331 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:05.081749 3108 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:04.931164 3333 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260927 23:31:05.042966 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.083343 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:05.083496 1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1468) took 1521 ms (client timeout 1704 ms). Trace:
I20260927 23:31:05.083972 1657 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:04.983485 3274 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:04.993988 2901 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:05.032804 1486 catalog_manager.cc:5495] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:05.049858 1551 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.051178 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:05.053561 1660 rpcz_store.cc:276] 0927 23:31:04.022628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.522616 (+499988us) service_pool.cc:227] Handling call
0927 23:31:04.853303 (+330687us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:05.052966 (+199663us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3969920}
W20260927 23:31:05.087881 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 118764us
I20260927 23:31:04.977736 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.066134 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 324) took 1090 ms. Trace:
I20260927 23:31:04.981817 3118 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:04.969856 3091 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.092054 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.966289 3311 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.072202 2289 rpcz_store.cc:276] 0927 23:31:03.797494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.075698 (+278204us) service_pool.cc:227] Handling call
0927 23:31:04.934555 (+858857us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:05.079824 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:05.033015 1585 raft_consensus.cc:603] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:04.999094 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:04.954577 1484 rpcz_store.cc:276] 0927 23:30:55.743106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:55.743722 (+ 616us) service_pool.cc:227] Handling call
0927 23:31:04.851755 (+9108033us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:00.971382 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:01.071203 (+ 99821us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5c002aff8c. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f9756df8501 00007f9756df83f1 00007f9756df312c 00007f9754492a59 00007f975447e603 00007f975447e2b8 00007f9754482ba1 00007f9754482b59 00007f9754482ae9 00007f9754482ab1 00007f9754481dad
0927 23:31:01.071297 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:01.071336 (+ 39us) write_op.cc:435] Acquired schema lock
0927 23:31:01.071355 (+ 19us) tablet.cc:660] Decoding operations
0927 23:31:01.071883 (+ 528us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:01.071979 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:01.072022 (+ 43us) tablet.cc:683] Acquiring locks for 2 operations
0927 23:31:01.072409 (+ 387us) tablet.cc:699] Row locks acquired
0927 23:31:01.072427 (+ 18us) write_op.cc:260] PREPARE: finished
0927 23:31:01.072574 (+ 147us) write_op.cc:270] Start()
0927 23:31:01.072806 (+ 232us) write_op.cc:276] Timestamp: P: 1790551861072525 usec, L: 0
0927 23:31:01.072836 (+ 30us) op_driver.cc:331] REPLICATION: starting
0927 23:31:01.096382 (+ 23546us) log.cc:842] Serialized 906 byte log entry
0927 23:31:02.119351 (+1022969us) op_driver.cc:444] REPLICATION: finished
0927 23:31:02.356089 (+236738us) write_op.cc:301] APPLY: starting
0927 23:31:02.370880 (+ 14791us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:02.406554 (+ 35674us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:02.409001 (+ 2447us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:02.498229 (+ 89228us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:02.514674 (+ 16445us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 23:31:02.514723 (+ 49us) write_op.cc:312] APPLY: finished
0927 23:31:02.514909 (+ 186us) log.cc:842] Serialized 29 byte log entry
0927 23:31:02.515055 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:02.522519 (+ 7464us) write_op.cc:454] Released schema lock
0927 23:31:02.526266 (+ 3747us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:02.532638 (+ 6372us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:02.535347 (+ 2709us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":269651,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":221332,"apply.run_cpu_time_us":2570,"apply.run_wall_time_us":190657,"num_ops":2,"prepare.queue_time_us":236047,"prepare.run_cpu_time_us":49390,"prepare.run_wall_time_us":281874,"raft.queue_time_us":737447,"raft.run_cpu_time_us":22274,"raft.run_wall_time_us":119670,"replication_time_us":1046422,"spinlock_wait_cycles":2794752,"thread_start_us":1330527,"threads_started":4,"wal-append.queue_time_us":194995,"wal-append.run_cpu_time_us":31551,"wal-append.run_wall_time_us":2189730}]]}
I20260927 23:31:05.089114 2291 rpcz_store.cc:276] 0927 23:31:03.976232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.347706 (+371474us) service_pool.cc:227] Handling call
0927 23:31:04.348069 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:04.349852 (+ 1783us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.349894 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:04.362986 (+ 13092us) log.cc:842] Serialized 26 byte log entry
0927 23:31:04.715236 (+352250us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:04.715289 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.715381 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:05.065910 (+350529us) raft_consensus.cc:1671] finished
0927 23:31:05.065934 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:05.066080 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1349,"raft.queue_time_us":1923,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":3611,"thread_start_us":273677,"threads_started":1,"wal-append.queue_time_us":369830}
I20260927 23:31:05.094825 1700 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.054600 3117 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.952891 3314 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.090770 1656 raft_consensus.cc:2396] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:05.086335 2287 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:05.054013 3135 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:04.977226 3145 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260927 23:31:05.052716 3206 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:04.978653 3247 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.100019 2294 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260927 23:31:05.100828 1552 leader_election.cc:398] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:05.115465 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.044061 2288 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:05.114162 3184 raft_consensus.cc:465] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Not starting pre-election -- already a leader
I20260927 23:31:05.117110 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:31:05.092425 3279 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.105140 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1178) took 1722 ms. Trace:
I20260927 23:31:05.105625 1657 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:05.107690 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.121925 1581 rpcz_store.cc:276] 0927 23:31:03.383134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.502783 (+1119649us) service_pool.cc:227] Handling call
0927 23:31:04.503034 (+ 251us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:04.772634 (+269600us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.104907 (+332273us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540009f6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:05.105088 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356889472}
W20260927 23:31:05.083904 1728 rpcz_store.cc:269] 0927 23:31:03.562290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.696353 (+134063us) service_pool.cc:227] Handling call
0927 23:31:05.083435 (+1387082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:05.072527 3171 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:05.039033 2290 rpcz_store.cc:269] 0927 23:31:03.398607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.593902 (+195295us) service_pool.cc:227] Handling call
0927 23:31:04.896829 (+1302927us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:05.087167 3274 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:05.098990 1654 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:05.075804 1732 rpcz_store.cc:276] 0927 23:31:04.042705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.219933 (+177228us) service_pool.cc:227] Handling call
0927 23:31:05.075307 (+855374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:04.983453 3239 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.066875 3304 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.118286 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.052883 3214 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:05.120647 3128 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.100163 2910 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.122854 1656 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:05.123744 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.155104 1657 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.156292 1657 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:05.125823 3206 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:05.131319 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.133119 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:05.067080 3262 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.106830 3091 raft_consensus.cc:686] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:05.136399 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:05.159344 1737 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.135720 3224 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.117826 3135 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.112141 3117 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.099524 3331 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260927 23:31:05.114741 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.118950 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:05.176344 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:05.155652 1583 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:05.180490 1737 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:05.117168 2288 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 404) took 1766 ms (client timeout 2201 ms). Trace:
I20260927 23:31:05.175138 3091 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.181883 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:05.181756 2288 rpcz_store.cc:269] 0927 23:31:03.350702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:03.383626 (+ 32924us) service_pool.cc:227] Handling call
0927 23:31:04.536192 (+1152566us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:05.117111 (+580919us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70674688}
I20260927 23:31:05.182632 1737 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.147917 1660 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:05.149217 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:05.184373 3117 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.185201 1577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ddb51781fec46b194174de5f4406def with delete_type TABLET_DATA_TOMBSTONED (TS 0fe98f486bab4d0b9c421c145d83f1ce not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46514
I20260927 23:31:05.187012 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.187918 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1493) took 1110 ms. Trace:
I20260927 23:31:05.188259 1658 rpcz_store.cc:276] 0927 23:31:04.077668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.524249 (+446581us) service_pool.cc:227] Handling call
0927 23:31:05.186887 (+662638us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0fbce 000000000034aea9 00007f975593ba30 00007f9755a017b4 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:05.187857 (+ 970us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6770432}
I20260927 23:31:05.189472 1737 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:05.107151 2906 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260927 23:31:05.168920 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.190701 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.190986 1732 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:05.099344 3337 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260927 23:31:05.106886 3282 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260927 23:31:05.110446 2901 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 23:31:05.110486 3108 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:05.164239 1700 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.166659 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.161803 3315 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.142660 3054 consensus_queue.cc:1046] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260927 23:31:05.170029 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.167796 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.191816 1554 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:05.135452 3311 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260927 23:31:05.216342 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.145339 3312 consensus_queue.cc:1046] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260927 23:31:05.200862 1733 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:05.155815 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.141408 3171 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:05.208601 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 44447.7us
I20260927 23:31:05.208717 1585 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 1.
I20260927 23:31:05.166776 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.227596 1581 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:05.227784 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.228504 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.161753 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.171792 3219 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.214515 1579 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:05.155032 3304 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.173393 3104 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:05.217726 1737 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260927 23:31:05.218206 3164 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.689s user 0.007s sys 0.003s
I20260927 23:31:05.219955 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.191465 3128 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.225620 1659 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:05.210748 3108 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.230415 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:05.232630 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.234138 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.174520 3224 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.246860 2259 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.163532 3206 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.216997 3163 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:05.243286 1579 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.199049 3333 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260927 23:31:05.219880 2906 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.243836 1580 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:05.258447 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.275995 1579 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:05.284281 1585 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.243125 3108 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:05.205615 3279 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.226344 2910 raft_consensus.cc:686] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:05.267433 3354 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260927 23:31:05.266183 3163 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.231966 3145 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260927 23:31:05.258818 3143 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.260823 3104 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.304536 3143 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.305299 3104 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.306638 3143 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.307451 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:05.307868 3145 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:05.307940 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:05.309698 1628 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.228245 3214 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.316648 1628 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.317974 1628 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.285900 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.258621 2288 raft_consensus.cc:2362] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3. Re-sending same reply.
I20260927 23:31:05.258706 3128 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.290779 3352 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.267781 3135 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:05.324810 3318 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.847s user 0.003s sys 0.000s
I20260927 23:31:05.328154 2910 consensus_queue.cc:237] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:05.284386 3345 log.cc:924] Time spent T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20260927 23:31:05.319133 1628 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.320106 1702 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:05.266575 3125 consensus_meta.cc:287] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.603s user 0.006s sys 0.003s
I20260927 23:31:05.293716 3247 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:05.234030 3330 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260927 23:31:05.300689 3163 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:05.295221 2906 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:05.347968 3351 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260927 23:31:05.335035 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.308255 3145 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:05.314934 3164 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.319130 3280 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:05.320379 3311 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:05.325418 3054 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:05.313915 3283 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.349601 1552 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:05.342092 3350 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260927 23:31:05.365401 3054 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.371129 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:05.352847 1584 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.341142 3125 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:05.343535 3252 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.384764 1552 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.385511 3054 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.349994 3262 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:05.404490 1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1307) took 2287 ms (client timeout 2944 ms). Trace:
I20260927 23:31:05.354820 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:05.401021 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.401325 1736 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:05.368722 3171 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.355408 3311 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:05.382043 3164 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:05.405066 1588 rpcz_store.cc:269] 0927 23:31:03.116831 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.217325 (+1100494us) service_pool.cc:227] Handling call
0927 23:31:05.404279 (+1186954us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b540038efd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:05.404410 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":874933504}
I20260927 23:31:05.417768 3262 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:05.418895 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.419101 3311 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:05.420022 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.421006 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.421923 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.422873 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.423625 1585 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.424249 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.425155 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.427438 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.428471 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.429343 1588 raft_consensus.cc:2396] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260927 23:31:05.401674 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 171263us
I20260927 23:31:05.430192 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46514 (request call id 854) took 2049 ms. Trace:
I20260927 23:31:05.430568 1586 rpcz_store.cc:276] 0927 23:31:03.381355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.579412 (+1198057us) service_pool.cc:227] Handling call
0927 23:31:05.429664 (+850252us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b5400308ed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4af0 00007f9750a77f2a 00007f97559b4c6e 00007f974f0ab807 00007f974f0ab792 00007f974f0ab6c1 00007f974f0ab63c 00007f974f0aa8f2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:05.430102 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559343360}
I20260927 23:31:05.413398 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.389287 3358 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.416652 3361 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:31:05.403508 2901 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.430713 1700 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.390372 3143 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:05.387208 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.429975 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.426060 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.433679 1579 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.481560 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.481926 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.482163 1588 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:05.482640 1579 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:05.434783 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 115733us
I20260927 23:31:05.432833 3304 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.435094 3360 tablet_replica.cc:330] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: stopping tablet replica
I20260927 23:31:05.436619 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.406222 3125 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.348487 2906 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.404623 3252 raft_consensus.cc:686] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.483400 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.416826 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 380) took 1911 ms. Trace:
I20260927 23:31:05.490839 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.433745 3315 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:05.479562 3206 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:05.429932 3311 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.481155 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.410710 3145 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.490474 1730 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
W20260927 23:31:05.548194 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64882.2us
I20260927 23:31:05.505132 1588 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:05.576013 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.576959 1588 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:05.561147 1586 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:05.579694 1586 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:05.495673 3224 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.490794 3104 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:05.504762 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:05.497085 1655 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.595s user 0.009s sys 0.002s
I20260927 23:31:05.561949 3128 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.557623 3369 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:05.590406 1588 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:05.553855 3117 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:05.593134 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.532825 3125 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260927 23:31:05.553604 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:05.563066 3283 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.505498 3312 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:05.580973 3206 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.589941 3164 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.604337 1588 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260927 23:31:05.615672 1588 pending_rounds.cc:76] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 5
I20260927 23:31:05.616050 1588 pending_rounds.cc:97] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260927 23:31:05.616441 1588 raft_consensus.cc:2866] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:05.591440 3104 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:05.603744 3352 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:05.600683 3177 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.601078 3369 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.604990 3171 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:05.630231 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 197973us
I20260927 23:31:05.604285 3311 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.601930 1655 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:05.555882 3252 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.642619 1702 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.622761 3104 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.604243 3108 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.652436 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.658064 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:05.658386 1730 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:05.658675 1733 raft_consensus.cc:2362] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3. Re-sending same reply.
I20260927 23:31:05.630771 3352 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.602361 3117 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.609817 3163 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.619656 3206 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.606505 3054 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.622541 3164 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.639587 3369 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:05.503995 2286 rpcz_store.cc:276] 0927 23:31:03.505625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.098415 (+592790us) service_pool.cc:227] Handling call
0927 23:31:04.098837 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.529749 (+430912us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.533229 (+ 3480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.535822 (+ 2593us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:04.542034 (+ 6212us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.416573 (+874539us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400541354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:05.416622 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:05.416770 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":947697536}
I20260927 23:31:05.670882 1735 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:05.673152 3054 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.678854 3164 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:05.680325 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:05.681025 3372 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260927 23:31:05.681219 1583 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:05.665865 3352 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:05.683425 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:05.638473 3177 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.650597 3171 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.693766 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.687929 3163 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.689841 3177 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.690279 3126 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:05.702678 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.687096 3280 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.695886 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.701001 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.703260 1733 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:05.704833 1730 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:05.683377 3135 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.707657 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.708240 3360 raft_consensus.cc:2229] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:31:05.709035 3360 raft_consensus.cc:2258] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:31:05.708894 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.710402 3280 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:05.720407 3323 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.030s user 0.003s sys 0.000s
I20260927 23:31:05.721875 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1457) took 2355 ms. Trace:
I20260927 23:31:05.699200 3304 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.722201 1729 rpcz_store.cc:276] 0927 23:31:03.366563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.236943 (+870380us) service_pool.cc:227] Handling call
0927 23:31:04.237272 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:04.237701 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.237740 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:04.248058 (+ 10318us) log.cc:842] Serialized 26 byte log entry
0927 23:31:04.479335 (+231277us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:04.479387 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:04.479476 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:05.719911 (+1240435us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:05.721597 (+ 1686us) raft_consensus.cc:1671] finished
0927 23:31:05.721614 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:05.721823 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33596,"raft.queue_time_us":1158,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":69629,"spinlock_wait_cycles":23074048,"thread_start_us":152028,"threads_started":1,"wal-append.queue_time_us":272523}
I20260927 23:31:05.724465 3171 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.725281 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 191) took 1626 ms. Trace:
I20260927 23:31:05.727893 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.729599 3280 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:05.731647 1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1498) took 1515 ms (client timeout 1752 ms). Trace:
W20260927 23:31:05.732033 1662 rpcz_store.cc:269] 0927 23:31:04.215912 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.503616 (+287704us) service_pool.cc:227] Handling call
0927 23:31:05.731475 (+1227859us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54002dc8d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:05.731598 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":690599808}
I20260927 23:31:05.725677 1587 rpcz_store.cc:276] 0927 23:31:04.098750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.523481 (+424731us) service_pool.cc:227] Handling call
0927 23:31:04.523820 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.933589 (+409769us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:04.933621 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:04.933655 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:04.933683 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.725082 (+791399us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001ff4d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:05.725124 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:05.725232 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":941899520}
I20260927 23:31:05.703044 3126 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:05.734215 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.734898 3126 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.735981 3126 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.740958 3239 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.741609 3117 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.751478 1702 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:05.756693 3206 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.757432 3304 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:05.746906 3135 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:05.758287 3117 consensus_queue.cc:1046] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:05.758680 3369 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.759262 3117 raft_consensus.cc:3032] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:05.706945 3163 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.759581 3117 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:05.760648 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 383) took 1664 ms. Trace:
I20260927 23:31:05.761179 3369 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:05.760690 3117 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.762288 3117 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:05.764118 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.764930 3108 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.764052 3104 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.766085 1731 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:05.766877 3104 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:05.767333 3360 ts_tablet_manager.cc:1905] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:31:05.761016 2293 rpcz_store.cc:276] 0927 23:31:04.096491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.422654 (+326163us) service_pool.cc:227] Handling call
0927 23:31:04.422927 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:04.423122 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:04.645789 (+222667us) log.cc:842] Serialized 14 byte log entry
0927 23:31:05.760263 (+1114474us) spinlock_profiling.cc:243] Waited 7.42 ms on lock 0x7b18002aeb00. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db44155 00007f9750a673d0 00007f9750a79576 00007f9750a8a0d7
0927 23:31:05.760354 (+ 91us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:05.760397 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:05.760436 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.760497 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:05.760598 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5110,"prepare.queue_time_us":344666,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":141385,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5627,"spinlock_wait_cycles":17071488,"thread_start_us":364716,"threads_started":2}
I20260927 23:31:05.779237 1661 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260927 23:31:05.779688 1661 pending_rounds.cc:76] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 5
I20260927 23:31:05.779928 1661 pending_rounds.cc:97] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260927 23:31:05.780194 1661 raft_consensus.cc:2866] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:05.754122 3280 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.787819 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.790059 1580 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:05.772123 3224 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:05.792006 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.792963 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.798189 1733 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 23:31:05.793053 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.801213 1628 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:05.802769 1733 raft_consensus.cc:1261] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:05.810103 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.799803 1660 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:05.806293 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:05.819693 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:05.824201 3274 consensus_meta.cc:287] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.672s user 0.007s sys 0.004s
I20260927 23:31:05.825174 3274 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:05.826725 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.827618 2291 raft_consensus.cc:1226] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:05.827821 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:05.832446 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:05.836328 1551 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:05.834362 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.838419 1587 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:05.841668 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:05.787734 3143 raft_consensus.cc:686] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:05.843459 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:05.828173 3282 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:05.850993 3378 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:05.788554 3108 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20260927 23:31:05.853294 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:05.855036 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:05.791741 3164 consensus_queue.cc:1046] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260927 23:31:05.857837 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:05.834033 3314 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:05.837212 3301 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:05.834164 3311 raft_consensus.cc:2741] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260927 23:31:05.817564 3135 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260927 23:31:05.826423 3224 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.892385 3333 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260927 23:31:05.854539 3378 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.851859 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:05.855157 3282 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:05.857261 2905 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:05.968921 3176 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:05.854588 3281 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:05.856835 3184 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.015111 3224 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.009311 3378 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:05.996650 3301 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:06.062495 3362 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260927 23:31:06.027249 3176 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:06.064257 3176 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.065907 3176 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:06.063135 3327 raft_consensus.cc:3032] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:06.067574 3327 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:06.028108 3282 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:06.068611 3327 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.069897 3327 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:06.067205 3324 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.406s user 0.003s sys 0.002s
I20260927 23:31:06.030019 3184 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.072695 3184 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.028775 3281 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.077912 3098 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:06.078436 3098 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.082844 3352 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.085512 3352 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.073654 3143 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.086851 3145 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:06.073583 3301 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:06.098568 3379 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260927 23:31:06.112421 3281 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.027113 2905 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:06.231329 2905 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:06.237911 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 380171us
I20260927 23:31:05.845468 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:06.238742 3117 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:06.240590 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 395052us
I20260927 23:31:06.240752 3061 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:06.166909 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:06.242236 3184 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.242718 3118 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.243253 3184 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.243527 3118 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.244591 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:06.245852 1583 raft_consensus.cc:2396] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:05.931277 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:06.246299 3184 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:06.246668 3177 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.246922 1660 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:06.247666 3177 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.173069 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1397) took 1076 ms. Trace:
I20260927 23:31:06.252319 1584 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 23:31:06.250486 1656 rpcz_store.cc:276] 0927 23:31:05.096273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.165389 (+ 69116us) service_pool.cc:227] Handling call
0927 23:31:05.165716 (+ 327us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.166066 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.166111 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.167569 (+ 1458us) log.cc:842] Serialized 26 byte log entry
0927 23:31:05.298152 (+130583us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:05.319922 (+ 21770us) log.cc:842] Serialized 14 byte log entry
0927 23:31:05.761931 (+442009us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.762070 (+ 139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:05.969739 (+207669us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b18003a7980. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:06.172223 (+202484us) raft_consensus.cc:1671] finished
0927 23:31:06.172389 (+ 166us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.172810 (+ 421us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":574,"prepare.queue_time_us":23410,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":24882,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8494,"spinlock_wait_cycles":13171200,"thread_start_us":514818,"threads_started":2,"wal-append.queue_time_us":132636}
I20260927 23:31:06.178359 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.255761 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:06.256189 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 1.82322e+06us
I20260927 23:31:06.257064 3126 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:06.257347 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.257674 3126 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:05.859179 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.259039 3274 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.259101 3128 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:31:06.259953 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:06.262831 3145 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.264060 3145 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.264896 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.265920 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1389) took 1425 ms. Trace:
I20260927 23:31:06.268074 3333 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:06.207201 2293 raft_consensus.cc:2354] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:06.127321 3378 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:05.901916 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:06.239689 3117 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.241359 3164 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.172295 1661 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:05.853253 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 422584us
I20260927 23:31:06.317036 3378 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.317826 1700 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:06.249910 3052 raft_consensus.cc:3032] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:06.272439 1551 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:06.269584 1663 rpcz_store.cc:276] 0927 23:31:04.840980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.264336 (+1423356us) service_pool.cc:227] Handling call
0927 23:31:06.265860 (+ 1524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:06.233898 3281 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:06.279865 1627 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:06.258205 3126 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:06.268591 3333 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:06.246697 3283 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.319089 1587 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:06.254115 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.313812 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 388) took 1308 ms. Trace:
I20260927 23:31:06.320111 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:06.326756 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 84496.7us
I20260927 23:31:06.328132 3175 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:06.328747 3175 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:06.330647 3283 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.331127 3199 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:06.331799 1656 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 23:31:06.333097 3283 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.334846 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:06.334612 3333 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.335649 1583 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:06.336704 3333 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:06.336989 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:06.345837 3365 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.931s user 0.003s sys 0.000s
I20260927 23:31:06.354027 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:06.312996 3118 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:06.321228 3052 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:06.330109 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:06.347697 3177 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.324739 2291 rpcz_store.cc:276] 0927 23:31:05.003809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.095056 (+ 91247us) service_pool.cc:227] Handling call
0927 23:31:05.095320 (+ 264us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:05.828052 (+732732us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.311543 (+483491us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b540071f2d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:06.311876 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1683662080}
I20260927 23:31:06.347383 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:06.335673 3312 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:06.386152 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:06.421566 3314 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:31:06.421769 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 91815.9us
W20260927 23:31:06.422610 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 153123us
I20260927 23:31:06.425858 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:06.427663 2901 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.916s user 0.023s sys 0.074s
I20260927 23:31:06.426276 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:06.429466 3279 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.026s sys 0.070s
I20260927 23:31:06.444219 3117 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.414258 3281 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:06.414194 1583 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:06.448040 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:06.430156 2288 raft_consensus.cc:2396] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:06.408537 3312 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:06.465539 1657 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:06.456948 3371 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:06.456715 3052 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.468024 3052 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:06.468063 1659 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:06.469976 1659 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 23:31:06.455951 1655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1485) took 2422 ms (client timeout 2245 ms). Trace:
W20260927 23:31:06.428879 2901 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.938s user 0.024s sys 0.075s
I20260927 23:31:06.455471 3341 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:06.456199 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1480) took 1328 ms. Trace:
W20260927 23:31:06.541468 1655 rpcz_store.cc:269] 0927 23:31:04.033435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.661670 (+628235us) service_pool.cc:227] Handling call
0927 23:31:06.455504 (+1793834us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b5400041ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:06.455721 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":163558144}
I20260927 23:31:06.544296 1731 rpcz_store.cc:276] 0927 23:31:05.127472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.132768 (+ 5296us) service_pool.cc:227] Handling call
0927 23:31:05.736808 (+604040us) spinlock_profiling.cc:243] Waited 8.78 ms on lock 0x7b5800282098. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:05.737001 (+ 193us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:05.766495 (+ 29494us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.455662 (+689167us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b5400212ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:06.456025 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86594176}
I20260927 23:31:06.545780 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:06.475385 3312 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.476933 1661 raft_consensus.cc:1261] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 23:31:06.486714 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 169195us
I20260927 23:31:06.474164 1627 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:06.487649 3373 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.492s user 0.000s sys 0.001s
I20260927 23:31:06.507736 3282 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.509353 2905 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:06.510219 1579 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.012s sys 0.030s
W20260927 23:31:06.464418 3279 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.027s sys 0.070s
I20260927 23:31:06.511603 1584 raft_consensus.cc:2396] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:06.516074 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1481) took 1388 ms. Trace:
I20260927 23:31:06.546777 1583 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.527506 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1385) took 1687 ms. Trace:
I20260927 23:31:06.546454 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:06.427811 2290 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 23:31:06.539443 1660 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:06.456598 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:06.471369 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1193) took 1306 ms. Trace:
W20260927 23:31:06.549784 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 92977.2us
I20260927 23:31:06.589954 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:06.410938 3164 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.567941 1735 rpcz_store.cc:276] 0927 23:31:05.127786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.127950 (+ 164us) service_pool.cc:227] Handling call
0927 23:31:05.128200 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:05.671733 (+543533us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.515576 (+843843us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400147b54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:06.515895 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1247486336}
W20260927 23:31:06.570142 1588 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.931s user 0.023s sys 0.027s
W20260927 23:31:06.581384 1579 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s user 0.013s sys 0.030s
I20260927 23:31:06.517354 3145 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.568473 1657 rpcz_store.cc:276] 0927 23:31:04.839799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.157053 (+317254us) service_pool.cc:227] Handling call
0927 23:31:06.465058 (+1308005us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:06.466154 (+ 1096us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.527179 (+ 61025us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400041ad0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:06.527410 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3002993536}
I20260927 23:31:06.547680 2901 raft_consensus.cc:686] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:06.557142 3281 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:06.589087 3052 raft_consensus.cc:2741] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:31:06.613479 1733 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:06.604627 1588 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.965s user 0.023s sys 0.027s
W20260927 23:31:06.605015 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 179421us
I20260927 23:31:06.619421 1661 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 23:31:06.626330 1657 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260927 23:31:06.626854 1657 pending_rounds.cc:76] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 4
I20260927 23:31:06.624805 1662 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:06.633481 1659 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.636713 1731 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:06.638520 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 164475us
I20260927 23:31:06.639492 1627 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:06.639235 2901 consensus_queue.cc:237] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.640797 1733 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
W20260927 23:31:06.641873 3098 consensus_meta.cc:287] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.563s user 0.001s sys 0.012s
I20260927 23:31:06.643030 1733 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:06.644019 3117 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.644771 3117 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:06.646412 2906 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.008s sys 0.044s
W20260927 23:31:06.646925 2906 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s user 0.008s sys 0.044s
W20260927 23:31:06.647941 3252 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.001s sys 0.005s
W20260927 23:31:06.648100 3301 consensus_meta.cc:287] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.553s user 0.004s sys 0.012s
W20260927 23:31:06.648398 3252 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s user 0.001s sys 0.005s
I20260927 23:31:06.648684 3301 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.631536 2905 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:06.652262 1586 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.010s sys 0.026s
W20260927 23:31:06.652710 1586 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.035s user 0.010s sys 0.026s
I20260927 23:31:06.592556 1582 rpcz_store.cc:276] 0927 23:31:05.164828 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.168232 (+ 3404us) service_pool.cc:227] Handling call
0927 23:31:05.170840 (+ 2608us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.235255 (+ 64415us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.235285 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.247919 (+ 12634us) log.cc:842] Serialized 26 byte log entry
0927 23:31:05.324367 (+ 76448us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:05.324411 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:05.324494 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:06.344697 (+1020203us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:06.470777 (+126080us) raft_consensus.cc:1671] finished
0927 23:31:06.470807 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.471136 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":714,"raft.queue_time_us":22482,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":14202,"spinlock_wait_cycles":39633152,"thread_start_us":99811,"threads_started":2,"wal-append.queue_time_us":62346}
W20260927 23:31:06.621542 3358 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.946s user 0.015s sys 0.088s
W20260927 23:31:06.620132 3214 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.840s user 0.021s sys 0.039s
I20260927 23:31:06.642087 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:06.627192 1657 pending_rounds.cc:97] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260927 23:31:06.662834 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 23:31:06.664611 1728 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.967s user 0.038s sys 0.044s
W20260927 23:31:06.665050 1728 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.033s user 0.038s sys 0.044s
W20260927 23:31:06.665483 3361 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.075s user 0.015s sys 0.035s
W20260927 23:31:06.665956 3361 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.162s user 0.015s sys 0.035s
I20260927 23:31:06.666353 3361 raft_consensus.cc:686] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:06.667428 3145 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.667827 1735 raft_consensus.cc:1261] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:06.668018 3239 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:06.668495 3239 raft_consensus.cc:686] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:06.668911 3361 consensus_queue.cc:237] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:06.670042 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.941s user 0.016s sys 0.044s
W20260927 23:31:06.670392 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.030s user 0.016s sys 0.044s
I20260927 23:31:06.670269 3274 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:06.670854 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:06.671039 3274 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.668030 3145 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.638023 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:06.669588 3239 consensus_queue.cc:237] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.669726 1584 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:06.661162 1657 raft_consensus.cc:2866] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:06.678018 1587 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:06.684137 1587 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
W20260927 23:31:06.686822 3054 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.010s sys 0.013s
W20260927 23:31:06.687446 3054 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s user 0.010s sys 0.013s
I20260927 23:31:06.672855 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:06.694904 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:06.666118 2294 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.017s sys 0.017s
I20260927 23:31:06.703567 3134 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.704294 3134 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.706681 1660 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:06.707726 1629 leader_election.cc:398] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:06.708133 1629 leader_election.cc:401] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:06.614286 3371 consensus_queue.cc:1046] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
W20260927 23:31:06.738835 2294 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s user 0.017s sys 0.017s
W20260927 23:31:06.697902 3214 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.124s user 0.022s sys 0.039s
I20260927 23:31:06.603909 3341 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.686232 1587 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:06.731473 3358 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.212s user 0.015s sys 0.089s
W20260927 23:31:06.749023 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 128199us
I20260927 23:31:06.746879 1659 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:06.745877 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:06.736271 2906 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.659030 3098 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.760066 1735 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.768782 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:06.769665 3098 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.770399 3098 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:06.770717 3388 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
W20260927 23:31:06.773277 3379 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.003s sys 0.026s
W20260927 23:31:06.773757 3379 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s user 0.003s sys 0.026s
I20260927 23:31:06.774706 3134 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.775161 3247 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:06.778165 3279 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:06.778620 3145 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.741941 3054 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:06.748147 3312 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.782399 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 197) took 1565 ms. Trace:
I20260927 23:31:06.786101 2291 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:06.784763 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 161623us
I20260927 23:31:06.747705 2905 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:06.766763 3282 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.770742 3098 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:06.752269 3358 raft_consensus.cc:686] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:06.794217 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:06.813766 1735 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:06.786208 1579 rpcz_store.cc:276] 0927 23:31:05.216762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.598161 (+381399us) service_pool.cc:227] Handling call
0927 23:31:05.598487 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.598835 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.598872 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.599297 (+ 425us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.650391 (+1051094us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:06.650459 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.650552 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:06.779713 (+129161us) raft_consensus.cc:1671] finished
0927 23:31:06.779730 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.782287 (+ 2557us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":522,"raft.queue_time_us":13806,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5119,"thread_start_us":995880,"threads_started":1,"wal-append.queue_time_us":1062915}
I20260927 23:31:06.799021 3098 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:06.796226 3301 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.779044 3175 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.806851 3117 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:06.820128 2906 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.828936 1583 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:06.829674 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1336) took 1042 ms. Trace:
I20260927 23:31:06.830052 1583 rpcz_store.cc:276] 0927 23:31:05.787211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.447654 (+660443us) service_pool.cc:227] Handling call
0927 23:31:06.829614 (+381960us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:06.828986 1729 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.027s sys 0.045s
W20260927 23:31:06.831338 1729 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.083s user 0.028s sys 0.045s
I20260927 23:31:06.824870 1731 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:31:06.837733 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 206651us
I20260927 23:31:06.831336 2906 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:06.831976 3164 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.877297 1551 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:06.852425 3337 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.953s user 0.030s sys 0.080s
I20260927 23:31:06.841427 3145 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:06.886282 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:06.887014 1628 leader_election.cc:334] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.673s (SENT)
I20260927 23:31:06.897835 1628 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:06.843814 3279 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.050s
I20260927 23:31:06.879261 3274 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:06.914804 3146 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.151s user 0.000s sys 0.040s
I20260927 23:31:06.838864 3054 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:06.897460 3337 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.998s user 0.031s sys 0.080s
I20260927 23:31:06.886619 2906 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:06.914821 3118 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:06.928462 1732 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.014s sys 0.022s
W20260927 23:31:06.929386 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 143412us
I20260927 23:31:06.930186 3301 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:06.934442 1734 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.009s sys 0.021s
I20260927 23:31:06.935043 3314 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 23:31:06.918547 3146 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.227s user 0.000s sys 0.041s
I20260927 23:31:06.941021 3054 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.945132 3358 consensus_queue.cc:237] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:06.897863 3352 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:06.926405 3126 consensus_meta.cc:287] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.541s user 0.014s sys 0.000s
I20260927 23:31:06.933784 3360 ts_tablet_manager.cc:1918] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:31:06.924999 3145 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.007237 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 198) took 1720 ms. Trace:
I20260927 23:31:06.990558 2294 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 389) took 1868 ms. Trace:
I20260927 23:31:07.023490 3279 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.024222 3279 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.023696 1588 rpcz_store.cc:276] 0927 23:31:05.286664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.617490 (+330826us) service_pool.cc:227] Handling call
0927 23:31:05.618121 (+ 631us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.618609 (+ 488us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.618649 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.619101 (+ 452us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.708535 (+1089434us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:06.742375 (+ 33840us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.751120 (+ 8745us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.006904 (+255784us) raft_consensus.cc:1671] finished
0927 23:31:07.006934 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.007148 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5159,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":5981,"thread_start_us":976709,"threads_started":1,"wal-append.queue_time_us":1260585}
I20260927 23:31:06.993587 3072 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:31:06.994592 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1201) took 1812 ms (client timeout 1673 ms). Trace:
W20260927 23:31:07.026595 1581 rpcz_store.cc:269] 0927 23:31:05.174835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.228260 (+ 53425us) service_pool.cc:227] Handling call
0927 23:31:06.955504 (+1727244us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400330ed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:06.986813 (+ 31309us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3271482240}
I20260927 23:31:07.028115 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.029417 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.029650 3312 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.030045 3176 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.030500 3247 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:07.030401 3312 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.030769 3176 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.031356 3247 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.032862 3176 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.032959 3312 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.033459 3247 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:06.964720 3341 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.029409 3164 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.035159 3164 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:06.995358 3386 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260927 23:31:07.037072 3164 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:06.968644 1732 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s user 0.015s sys 0.023s
I20260927 23:31:06.996274 3301 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:06.970626 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 197) took 1231 ms. Trace:
I20260927 23:31:06.993471 3274 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:06.991925 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 105753us
I20260927 23:31:06.995788 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1322) took 1786 ms. Trace:
I20260927 23:31:06.964954 3118 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:06.968261 1658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.008s sys 0.041s
I20260927 23:31:07.030704 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.022934 2294 rpcz_store.cc:276] 0927 23:31:05.121701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.121838 (+ 137us) service_pool.cc:227] Handling call
0927 23:31:05.122185 (+ 347us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.212316 (+ 90131us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.212343 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.693260 (+480917us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b2c00151ea8. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:05.698435 (+ 5175us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.772760 (+1074325us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:06.772823 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.772963 (+ 140us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:06.954710 (+181747us) raft_consensus.cc:1671] finished
0927 23:31:06.954738 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.990457 (+ 35719us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11926,"raft.queue_time_us":27479,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":7188,"spinlock_wait_cycles":7811456,"thread_start_us":1132278,"threads_started":2,"wal-append.queue_time_us":1247866}
W20260927 23:31:07.051622 1734 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s user 0.009s sys 0.032s
I20260927 23:31:07.030189 1731 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:07.053378 3061 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:07.053506 1658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s user 0.009s sys 0.042s
I20260927 23:31:07.039007 3314 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:07.068679 1736 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.005s sys 0.029s
W20260927 23:31:07.076862 1736 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.005s sys 0.029s
I20260927 23:31:07.072826 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.071360 3301 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.053567 3118 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:07.057267 3360 log.cc:1196] T 5ddb51781fec46b194174de5f4406def P 0fe98f486bab4d0b9c421c145d83f1ce: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-0-root/wals/5ddb51781fec46b194174de5f4406def
I20260927 23:31:07.080087 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.050640 1729 rpcz_store.cc:276] 0927 23:31:05.738841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.739126 (+ 285us) service_pool.cc:227] Handling call
0927 23:31:05.739495 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.747482 (+ 7987us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.747524 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:05.748239 (+ 715us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.894308 (+1146069us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:06.897686 (+ 3378us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.925349 (+ 27663us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:06.954880 (+ 29531us) raft_consensus.cc:1671] finished
0927 23:31:06.954905 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.970564 (+ 15659us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7575,"raft.queue_time_us":9280,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":210,"thread_start_us":1087067,"threads_started":1,"wal-append.queue_time_us":1200882}
I20260927 23:31:07.101284 1657 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:07.107982 3143 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.075s sys 0.097s
W20260927 23:31:07.108513 3143 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.019s user 0.075s sys 0.097s
I20260927 23:31:07.109210 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1512) took 1171 ms. Trace:
I20260927 23:31:07.109754 1658 rpcz_store.cc:276] 0927 23:31:05.938253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.174695 (+236442us) service_pool.cc:227] Handling call
0927 23:31:06.175590 (+ 895us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:06.176405 (+ 815us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:06.176449 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.177136 (+ 687us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.093339 (+916203us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:07.100380 (+ 7041us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.108862 (+ 8482us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.108937 (+ 75us) raft_consensus.cc:1671] finished
0927 23:31:07.108960 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.109139 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73063,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"thread_start_us":895466,"threads_started":1,"wal-append.queue_time_us":820196}
I20260927 23:31:07.112282 3279 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:07.022586 3126 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:07.123216 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 61518.3us
I20260927 23:31:07.052067 1586 rpcz_store.cc:276] 0927 23:31:05.209609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.580687 (+371078us) service_pool.cc:227] Handling call
0927 23:31:05.597971 (+ 17284us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:05.598124 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:06.653122 (+1054998us) log.cc:842] Serialized 14 byte log entry
0927 23:31:06.992874 (+339752us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:06.992968 (+ 94us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:06.993172 (+ 204us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.993286 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:06.995738 (+ 2452us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1401418,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":21980,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":1341174,"threads_started":2}
W20260927 23:31:07.129822 1663 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.016s sys 0.034s
W20260927 23:31:07.130141 1663 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.016s sys 0.034s
I20260927 23:31:07.127683 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:07.132814 1587 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:07.117575 3341 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.142784 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.059237 3352 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.142134 3146 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260927 23:31:07.146695 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:07.147462 2290 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:07.148712 3387 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.149401 3387 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.152101 3387 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.146788 3392 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.075891 3274 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.100725 3054 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
I20260927 23:31:07.164664 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.125058 3098 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.174527 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:07.121747 3061 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.163043 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 195) took 1655 ms. Trace:
I20260927 23:31:07.128149 3314 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.167930 3052 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:07.169994 3387 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:07.223258 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 62667.3us
I20260927 23:31:07.223451 3360 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46514 (request call id 855) took 2092 ms. Trace:
I20260927 23:31:07.223901 3360 rpcz_store.cc:276] 0927 23:31:05.152392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.184888 (+ 32496us) service_pool.cc:227] Handling call
0927 23:31:05.352973 (+168085us) ts_tablet_manager.cc:1145] Deleting tablet 5ddb51781fec46b194174de5f4406def
0927 23:31:05.393697 (+ 40724us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0927 23:31:06.109140 (+715443us) tablet_metadata.cc:641] Metadata flushed
0927 23:31:06.933122 (+823982us) tablet_metadata.cc:641] Metadata flushed
0927 23:31:07.223331 (+290209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140222,"tablet-delete.queue_time_us":146274,"thread_start_us":71908,"threads_started":1}
I20260927 23:31:07.224671 1551 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:07.224977 3219 raft_consensus.cc:3032] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:07.238925 1657 raft_consensus.cc:2354] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:07.240854 3393 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:07.241933 3061 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.237996 3143 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.242866 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.243395 3143 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.244180 3282 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:07.244776 3282 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:07.244917 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.245570 3143 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.245755 3282 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.246364 3387 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.250914 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:07.253252 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.246147 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.215339 3145 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:07.218598 3252 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:07.174943 3341 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.178805 3392 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.244351 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.225383 3219 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:07.265718 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 595105us
W20260927 23:31:07.267613 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.005s sys 0.014s
I20260927 23:31:07.270566 1735 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:31:07.284135 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.006s sys 0.014s
I20260927 23:31:07.265199 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 194) took 1842 ms. Trace:
I20260927 23:31:07.250406 2906 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:07.278219 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 114054us
I20260927 23:31:07.220237 3117 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.248633 3175 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.280652 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:07.284519 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 576469us
I20260927 23:31:07.251674 3314 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.250391 3274 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.261749 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.299695 3252 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.301165 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.303556 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.284086 3214 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.291559 3145 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:07.354710 3145 raft_consensus.cc:686] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:07.355958 3145 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.300019 3219 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.320715 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.322077 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:07.322299 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1209) took 1536 ms (client timeout 1600 ms). Trace:
I20260927 23:31:07.322657 3117 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.323755 3239 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:07.324031 3404 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.202s user 0.000s sys 0.001s
I20260927 23:31:07.325505 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.326241 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.327128 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:07.328014 1586 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:07.328797 3392 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.329057 1584 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 23:31:07.308019 2906 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:07.373615 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.311116 3126 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.330730 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1520) took 1150 ms. Trace:
I20260927 23:31:07.331502 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 186) took 2512 ms. Trace:
W20260927 23:31:07.332407 3415 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 23:31:07.332744 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:07.334335 3412 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.116s user 0.000s sys 0.001s
I20260927 23:31:07.291350 1731 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
W20260927 23:31:07.384915 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 475096us
I20260927 23:31:07.386137 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.338970 3352 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.340590 3387 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.344995 3118 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.301036 3054 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.284638 1736 rpcz_store.cc:276] 0927 23:31:05.423093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.423272 (+ 179us) service_pool.cc:227] Handling call
0927 23:31:05.423596 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.423910 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.423937 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.531240 (+1107303us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b2c00018ac8. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:06.531919 (+ 679us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.077253 (+545334us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:07.079398 (+ 2145us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.079716 (+ 318us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.255688 (+175972us) raft_consensus.cc:1671] finished
0927 23:31:07.255714 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.265126 (+ 9412us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38842,"raft.queue_time_us":58977,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6669,"spinlock_wait_cycles":18228992,"thread_start_us":544907,"threads_started":1,"wal-append.queue_time_us":721824}
I20260927 23:31:07.348876 3214 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:07.363515 1580 rpcz_store.cc:269] 0927 23:31:05.785480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.790893 (+ 5413us) service_pool.cc:227] Handling call
0927 23:31:07.321960 (+1531067us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54003b42d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:07.322127 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3437772672}
I20260927 23:31:07.367445 3279 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.329851 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:07.374311 1662 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:07.377606 1581 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:07.306756 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1510) took 1353 ms. Trace:
I20260927 23:31:07.378397 1730 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:07.400224 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.382809 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.384861 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.387079 1587 raft_consensus.cc:2379] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:07.401024 1733 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:07.367866 3239 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:07.387614 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.393242 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.397761 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.404284 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.372534 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.374645 1656 rpcz_store.cc:276] 0927 23:31:06.180441 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.332846 (+152405us) service_pool.cc:227] Handling call
0927 23:31:06.333240 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:06.333854 (+ 614us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:06.333881 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.354631 (+ 20750us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.899100 (+544469us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:06.899170 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.899291 (+ 121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.326215 (+426924us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:07.330377 (+ 4162us) raft_consensus.cc:1671] finished
0927 23:31:07.330405 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.330661 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":352,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":45780736,"thread_start_us":483971,"threads_started":1,"wal-append.queue_time_us":837713}
I20260927 23:31:07.414785 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.416970 1662 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:07.384840 1472 catalog_manager.cc:4841] TS 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): tablet 5ddb51781fec46b194174de5f4406def (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:31:07.399616 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:07.376595 1734 rpcz_store.cc:276] 0927 23:31:04.818658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:04.818818 (+ 160us) service_pool.cc:227] Handling call
0927 23:31:04.819178 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:04.819750 (+ 572us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:04.864437 (+ 44687us) log.cc:842] Serialized 14 byte log entry
0927 23:31:05.973068 (+1108631us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b180033aba0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db29345 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f657b 00007f9750a7988a
0927 23:31:07.090290 (+1117222us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.145658 (+ 55368us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.171618 (+ 25960us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:07.171667 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.192401 (+ 20734us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.331208 (+138807us) raft_consensus.cc:1671] finished
0927 23:31:07.331307 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.331437 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":163,"prepare.queue_time_us":44302,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":6993,"raft.queue_time_us":20458,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":12040,"spinlock_wait_cycles":12561152,"thread_start_us":1081382,"threads_started":1,"wal-append.queue_time_us":2464565}
I20260927 23:31:07.419194 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.405018 1736 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:07.405646 1582 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:07.363977 3393 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260927 23:31:07.411113 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.413247 1735 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:07.421703 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.422628 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.423303 3117 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.423885 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.424568 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.426512 3219 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:07.464201 2290 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:07.466143 1655 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:07.467151 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:07.468793 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 78926.7us
I20260927 23:31:07.426880 3239 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.401975 1654 rpcz_store.cc:276] 0927 23:31:05.860044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.892310 (+ 32266us) service_pool.cc:227] Handling call
0927 23:31:06.171184 (+278874us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:06.174720 (+ 3536us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:06.174802 (+ 82us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.175751 (+ 949us) log.cc:842] Serialized 26 byte log entry
0927 23:31:06.595090 (+419339us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:07.161963 (+566873us) log.cc:842] Serialized 14 byte log entry
0927 23:31:07.167606 (+ 5643us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.167723 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.170499 (+ 2776us) raft_consensus.cc:1671] finished
0927 23:31:07.172279 (+ 1780us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.209381 (+ 37102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3134,"prepare.queue_time_us":664050,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":82189,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":2777,"thread_start_us":802736,"threads_started":2,"wal-append.queue_time_us":832300}
I20260927 23:31:07.414610 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.465241 2294 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260927 23:31:07.428871 3214 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.430112 3118 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.409056 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1499) took 1596 ms. Trace:
I20260927 23:31:07.403885 3054 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.414873 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.433776 1581 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:07.434909 1659 raft_consensus.cc:1261] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:07.435985 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.439847 1584 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:07.440605 3416 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:07.442049 1586 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:07.409684 1658 raft_consensus.cc:2396] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:07.453224 1582 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:07.440639 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:07.486611 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:07.459825 3380 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:07.460208 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1315) took 2207 ms (client timeout 2353 ms). Trace:
I20260927 23:31:07.460299 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.465791 2293 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.464994 1627 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:07.402240 3098 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:07.476923 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.482728 1730 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:07.491219 1585 rpcz_store.cc:269] 0927 23:31:05.201566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.425878 (+224312us) service_pool.cc:227] Handling call
0927 23:31:07.399079 (+1973201us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400308ed4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:07.409121 (+ 10042us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3743570688}
I20260927 23:31:07.483949 1662 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:07.483213 3416 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:07.497615 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:07.497836 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.498878 2294 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:07.475951 1733 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:07.496342 1734 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:07.503832 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1490) took 2184 ms. Trace:
I20260927 23:31:07.504472 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.505827 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.504350 1737 rpcz_store.cc:276] 0927 23:31:05.319242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.506260 (+1187018us) service_pool.cc:227] Handling call
0927 23:31:06.506858 (+ 598us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:06.590795 (+ 83937us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:06.590845 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:06.617106 (+ 26261us) log.cc:842] Serialized 52 byte log entry
0927 23:31:06.881979 (+264873us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:06.882019 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:06.882087 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.333776 (+451689us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:07.500069 (+166293us) raft_consensus.cc:1671] finished
0927 23:31:07.500093 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.503636 (+ 3543us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62117,"raft.queue_time_us":85783,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":51856,"spinlock_wait_cycles":34571520,"thread_start_us":242300,"threads_started":1,"wal-append.queue_time_us":522655}
I20260927 23:31:07.503832 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.507586 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.508939 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.511085 1658 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:07.508002 3416 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.516180 1654 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:07.497665 3301 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:07.507633 1736 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 5
I20260927 23:31:07.418861 3279 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.532787 1660 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:07.490814 3072 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:07.538002 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:07.531778 3337 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.027s sys 0.031s
I20260927 23:31:07.539135 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.565593 1662 raft_consensus.cc:2379] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 2: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:07.539398 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:07.540638 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.540892 3341 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:07.490608 3239 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.494601 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.558295 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:07.562809 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 83359.2us
I20260927 23:31:07.525899 1735 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:07.530028 2293 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:07.416724 3361 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:07.569900 1733 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:07.576226 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.581130 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.570822 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.586411 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.571492 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.572603 1730 raft_consensus.cc:2354] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
W20260927 23:31:07.569878 3401 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.011s sys 0.055s
W20260927 23:31:07.602790 3409 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
W20260927 23:31:07.602977 3401 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.598s user 0.011s sys 0.056s
I20260927 23:31:07.603480 3279 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.571178 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:07.495527 3387 consensus_queue.cc:1046] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260927 23:31:07.604925 3387 raft_consensus.cc:3032] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:07.605059 1658 raft_consensus.cc:1226] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:07.605165 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1514) took 1432 ms. Trace:
I20260927 23:31:07.605587 3387 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:07.605743 1663 rpcz_store.cc:276] 0927 23:31:06.172698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:06.324382 (+151684us) service_pool.cc:227] Handling call
0927 23:31:06.344107 (+ 19725us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:06.355913 (+ 11806us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:06.355951 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.356518 (+ 567us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.130517 (+773999us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:07.130561 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.139001 (+ 8440us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.604931 (+465930us) raft_consensus.cc:1671] finished
0927 23:31:07.604951 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.605097 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8168,"raft.queue_time_us":12348,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":16461,"thread_start_us":773596,"threads_started":1,"wal-append.queue_time_us":1017540}
I20260927 23:31:07.606673 3387 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.591208 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:07.569468 3337 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s user 0.027s sys 0.031s
I20260927 23:31:07.213649 1728 rpcz_store.cc:276] 0927 23:31:05.505076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.513269 (+ 8193us) service_pool.cc:227] Handling call
0927 23:31:05.571456 (+ 58187us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:05.593548 (+ 22092us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:06.665507 (+1071959us) log.cc:842] Serialized 14 byte log entry
0927 23:31:07.054715 (+389208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:07.060468 (+ 5753us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:07.128840 (+ 68372us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.138070 (+ 9230us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.160603 (+ 22533us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1323395,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":163,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3044,"thread_start_us":1410417,"threads_started":2,"wal-append.queue_time_us":589633}
I20260927 23:31:07.616195 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.602450 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:07.604004 3279 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:07.608620 1657 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:07.569373 3072 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.552721 3052 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260927 23:31:07.608270 3387 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:07.600251 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 135334us
I20260927 23:31:07.519809 2906 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.615302 3252 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.652859 1734 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:07.554719 3380 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.662931 3054 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:07.663844 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.664897 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.665426 3279 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.602181 1731 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:07.617014 3214 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:07.667667 3219 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:07.675462 3061 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 23:31:07.677629 3418 log.cc:924] Time spent T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
W20260927 23:31:07.678406 3405 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
I20260927 23:31:07.680313 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.681190 1657 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:07.664381 1551 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:07.691884 1660 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:07.704567 1662 raft_consensus.cc:1226] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:07.705935 1733 raft_consensus.cc:2396] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:07.709350 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.710651 1735 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:31:07.713867 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 49567.1us
I20260927 23:31:07.714432 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.695187 3214 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:07.726749 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 70831.9us
I20260927 23:31:07.736843 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.739785 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.726586 1700 leader_election.cc:398] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:07.741096 1700 leader_election.cc:401] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:07.742175 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.792021 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.703359 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:07.656343 3361 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:07.792744 1735 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:07.805048 1663 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260927 23:31:07.809465 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 339591us
I20260927 23:31:07.724856 3054 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:07.744411 3214 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:07.826463 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.826828 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.743692 3393 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:07.831286 1628 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:07.834097 1733 raft_consensus.cc:2396] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 23:31:07.480326 1732 rpcz_store.cc:276] 0927 23:31:05.807875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:05.817884 (+ 10009us) service_pool.cc:227] Handling call
0927 23:31:05.848600 (+ 30716us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:05.970949 (+122349us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:05.995958 (+ 25009us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:06.193521 (+197563us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.041426 (+847905us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:07.050231 (+ 8805us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.050402 (+ 171us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:07.392771 (+342369us) raft_consensus.cc:1671] finished
0927 23:31:07.392793 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:07.403817 (+ 11024us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38953,"raft.queue_time_us":74520,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":27854,"thread_start_us":844656,"threads_started":1,"wal-append.queue_time_us":878616}
I20260927 23:31:07.821880 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:07.847913 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.018s sys 0.061s
W20260927 23:31:07.849414 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.018s sys 0.061s
W20260927 23:31:07.849808 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 542998us
I20260927 23:31:07.854054 1733 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:07.823159 1658 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:07.834863 3301 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:07.809600 3252 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260927 23:31:07.836799 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:07.861892 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 122506us
W20260927 23:31:07.864490 1729 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.010s sys 0.027s
W20260927 23:31:07.850489 3282 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.009s sys 0.047s
W20260927 23:31:07.895318 1729 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.010s sys 0.027s
I20260927 23:31:07.863415 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:07.841759 3393 raft_consensus.cc:686] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:07.844185 3358 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s user 0.039s sys 0.079s
I20260927 23:31:07.852362 3117 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:07.847680 3426 raft_consensus.cc:2741] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 23:31:07.841215 3054 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:07.897873 3282 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s user 0.009s sys 0.047s
W20260927 23:31:07.912724 2257 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.016s sys 0.046s
W20260927 23:31:07.901005 3358 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.865s user 0.039s sys 0.080s
W20260927 23:31:07.915640 2257 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.017s sys 0.046s
W20260927 23:31:07.924362 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 601057us
I20260927 23:31:07.858855 3118 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
W20260927 23:31:07.937848 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 406183us
W20260927 23:31:07.939792 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 217666us
I20260927 23:31:07.904934 3117 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:07.911933 3052 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:07.928362 3426 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:07.936724 1731 raft_consensus.cc:2354] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:07.969897 1586 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:07.969820 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:07.971572 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:07.901783 3301 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:07.989682 3436 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:07.944520 1732 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:08.004542 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 109586us
I20260927 23:31:07.946995 3380 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:07.963941 3279 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:08.061180 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 89723.2us
W20260927 23:31:08.059500 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 230195us
I20260927 23:31:08.073343 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:08.074659 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 224569us
W20260927 23:31:08.061241 3175 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.736s user 0.011s sys 0.001s
W20260927 23:31:08.087229 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 55405.7us
I20260927 23:31:08.039906 2906 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:08.155874 1737 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:08.080174 3279 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:08.066393 3426 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:08.040060 3052 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.031946 3072 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.005498 3393 consensus_queue.cc:237] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.080736 3436 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.261270 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:08.500651 2258 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:08.260362 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.501509 3392 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:08.127980 3433 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:08.502578 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 429380us
W20260927 23:31:08.502943 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 779111us
W20260927 23:31:08.405673 3432 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20260927 23:31:08.504217 3445 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:08.506151 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1536) took 1207 ms. Trace:
I20260927 23:31:08.091786 3337 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:08.507263 3445 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:08.506702 1655 rpcz_store.cc:276] 0927 23:31:07.298386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.466939 (+168553us) service_pool.cc:227] Handling call
0927 23:31:07.467257 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.467636 (+ 379us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.467673 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.468163 (+ 490us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.829132 (+360969us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:07.832335 (+ 3203us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.843762 (+ 11427us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:08.505886 (+662124us) raft_consensus.cc:1671] finished
0927 23:31:08.505922 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.506082 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":325955,"threads_started":1,"wal-append.queue_time_us":504839}
I20260927 23:31:08.402935 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1528) took 1150 ms. Trace:
W20260927 23:31:08.367036 3435 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.331s user 0.000s sys 0.000s
I20260927 23:31:08.239464 1702 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:08.512465 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1345) took 1177 ms. Trace:
W20260927 23:31:08.511792 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 363067us
W20260927 23:31:08.513278 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 472963us
I20260927 23:31:08.512862 1588 rpcz_store.cc:276] 0927 23:31:07.335018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.468307 (+133289us) service_pool.cc:227] Handling call
0927 23:31:07.468767 (+ 460us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.469175 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.469216 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.469754 (+ 538us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.826573 (+356819us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:07.826610 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.826680 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:08.512259 (+685579us) raft_consensus.cc:1671] finished
0927 23:31:08.512278 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.512394 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9231,"raft.queue_time_us":4926,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":356417,"threads_started":1,"wal-append.queue_time_us":476750}
I20260927 23:31:08.514132 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:08.514889 3430 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 23:31:08.203914 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.516731 3098 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:08.520839 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:08.522130 3445 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:08.522697 3445 raft_consensus.cc:686] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:08.523725 3445 consensus_queue.cc:237] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.524950 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.525642 1583 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:08.526937 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.500916 1584 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:08.510720 1656 rpcz_store.cc:276] 0927 23:31:07.252712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.423476 (+170764us) service_pool.cc:227] Handling call
0927 23:31:07.423822 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.424263 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.424309 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.424853 (+ 544us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.845954 (+421101us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:07.845995 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.846060 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:08.402632 (+556572us) raft_consensus.cc:1671] finished
0927 23:31:08.402662 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.402855 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3629,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":420718,"threads_started":1,"wal-append.queue_time_us":907208}
W20260927 23:31:08.531317 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 329945us
I20260927 23:31:08.532976 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:08.534917 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.535130 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:08.535648 1584 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:08.501951 3392 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:08.162029 3361 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:08.543556 3431 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:08.127928 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.544458 3054 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:08.544658 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:08.544827 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 483872us
I20260927 23:31:08.545707 3468 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:08.572184 1700 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:08.546573 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 265378us
I20260927 23:31:08.546795 3452 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:08.600512 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.184948 3175 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:08.547572 3392 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:08.547425 3426 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:08.552698 1736 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.996s user 0.010s sys 0.000s
I20260927 23:31:08.611585 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:08.611913 1736 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:08.587988 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:08.612344 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 79458.6us
I20260927 23:31:08.598623 3337 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:08.598395 3450 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260927 23:31:08.599823 1627 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:08.598109 1730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.018s sys 0.022s
W20260927 23:31:08.629411 1730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.018s sys 0.023s
I20260927 23:31:08.127635 3252 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
W20260927 23:31:08.403710 3429 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.199s user 0.000s sys 0.000s
I20260927 23:31:08.637207 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1214) took 1292 ms. Trace:
I20260927 23:31:08.349462 3117 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:08.634244 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:08.639966 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:08.549757 3282 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:08.640620 1736 rpcz_store.cc:276] 0927 23:31:07.320293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.505589 (+185296us) service_pool.cc:227] Handling call
0927 23:31:08.612398 (+1106809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:08.639696 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1513) took 1254 ms. Trace:
W20260927 23:31:08.648492 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 48746.9us
I20260927 23:31:08.643147 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1228) took 1233 ms. Trace:
I20260927 23:31:08.651016 1735 rpcz_store.cc:276] 0927 23:31:07.409246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:08.080511 (+671265us) service_pool.cc:227] Handling call
0927 23:31:08.500161 (+419650us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:08.642717 (+142556us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:08.642760 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:08.642814 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:08.642855 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:08.642933 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.643085 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14719,"thread_start_us":83939,"threads_started":1}
I20260927 23:31:08.241503 2906 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.654826 2291 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:08.657153 1702 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:08.649609 1627 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:08.159251 3423 consensus_queue.cc:1046] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.448s
I20260927 23:31:08.631834 3380 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:08.607944 3436 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:08.648589 1730 rpcz_store.cc:276] 0927 23:31:07.385638 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.945214 (+559576us) service_pool.cc:227] Handling call
0927 23:31:07.991076 (+ 45862us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:08.639286 (+648210us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:08.639380 (+ 94us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:08.639427 (+ 47us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:08.639461 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:08.639532 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.639648 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122286,"raft.queue_time_us":295156,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":40525,"thread_start_us":604469,"threads_started":1}
I20260927 23:31:08.644590 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:08.670673 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:08.633958 3126 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:08.640336 3361 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:08.634644 3252 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:08.540150 3279 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:08.658072 3146 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.012s sys 0.064s
W20260927 23:31:08.674108 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 102044us
W20260927 23:31:08.684129 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 96266.2us
I20260927 23:31:08.705027 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:08.706480 1700 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:08.611658 3175 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.653479 3282 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:08.710552 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.660948 3301 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:08.716776 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59752.9us
I20260927 23:31:08.708185 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:08.722944 1736 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:08.660972 3454 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:08.731508 3439 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:08.731974 3449 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:08.671875 3052 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:08.630684 3468 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:08.733959 3416 consensus_meta.cc:287] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.186s user 0.012s sys 0.000s
I20260927 23:31:08.733655 3252 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.735595 3279 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:08.736617 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1508) took 1398 ms. Trace:
W20260927 23:31:08.709543 3146 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s user 0.012s sys 0.064s
I20260927 23:31:08.689383 3436 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:08.676090 3465 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:08.814903 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 108513us
I20260927 23:31:08.703262 3054 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:08.768112 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:08.739389 3358 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:08.722337 3282 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:08.753857 3301 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:08.799669 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1190) took 1558 ms. Trace:
I20260927 23:31:08.726046 3452 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:08.797201 1734 rpcz_store.cc:276] 0927 23:31:07.338651 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.859425 (+520774us) service_pool.cc:227] Handling call
0927 23:31:07.895062 (+ 35637us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:08.127870 (+232808us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:08.127909 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:08.159106 (+ 31197us) log.cc:842] Serialized 26 byte log entry
0927 23:31:08.511066 (+351960us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:08.511151 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:08.511346 (+ 195us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:08.736400 (+225054us) raft_consensus.cc:1671] finished
0927 23:31:08.736419 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.736561 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":253323,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":76083,"thread_start_us":357460,"threads_started":2,"wal-append.queue_time_us":473431}
W20260927 23:31:08.754565 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 120387us
W20260927 23:31:08.797724 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 72466.9us
W20260927 23:31:08.821163 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 149780us
I20260927 23:31:08.811100 3052 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:08.826220 1729 rpcz_store.cc:276] 0927 23:31:07.241035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.267210 (+ 26175us) service_pool.cc:227] Handling call
0927 23:31:07.283755 (+ 16545us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.292454 (+ 8699us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.292502 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.304409 (+ 11907us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.895825 (+591416us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:07.895878 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.895976 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:08.799405 (+903429us) raft_consensus.cc:1671] finished
0927 23:31:08.799433 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:08.799599 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":433,"raft.queue_time_us":1028,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7336,"thread_start_us":575418,"threads_started":1,"wal-append.queue_time_us":724402}
I20260927 23:31:08.837036 1627 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:08.799214 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.840682 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.820595 3054 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:08.803591 3416 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:08.868000 1627 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:08.814900 3279 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:08.754791 3126 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:08.825806 3301 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:08.874476 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:08.888895 1627 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:08.834230 3252 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:08.897022 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 99916us
I20260927 23:31:08.853983 3054 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:08.822909 3436 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.778023 3454 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:08.968178 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:08.865554 3471 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:08.888236 3452 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:31:08.972577 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 131944us
W20260927 23:31:08.973328 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 104197us
I20260927 23:31:08.811483 3449 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:08.979351 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 211464us
I20260927 23:31:08.807497 3439 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:08.900525 3301 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:08.920552 3337 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:08.991881 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:08.886902 3459 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:08.871943 3455 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:08.854326 3052 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:08.981066 3452 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:08.966531 3466 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:08.823161 3465 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:08.990482 3441 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.687s user 0.002s sys 0.000s
I20260927 23:31:08.979318 3146 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:09.041544 2259 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.006072 3467 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:09.015424 3337 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:09.015729 3387 consensus_meta.cc:287] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.332s user 0.004s sys 0.009s
I20260927 23:31:08.952484 3279 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.001617 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:09.015031 3448 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:09.052887 2259 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.041940 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:09.054931 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 444889us
W20260927 23:31:09.055757 3460 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.317s user 0.001s sys 0.001s
I20260927 23:31:09.056684 3472 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:09.057272 3478 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:31:09.058173 3438 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
I20260927 23:31:09.018036 3471 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:09.059640 3466 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:09.019776 2906 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.061251 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 349) took 1255 ms. Trace:
I20260927 23:31:09.046166 3452 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:09.134276 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:09.103919 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 262723us
I20260927 23:31:09.042061 3175 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:09.045658 3455 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:09.106739 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 401744us
I20260927 23:31:09.116451 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:09.115846 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 283078us
I20260927 23:31:09.048450 3358 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:09.126946 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 149440us
W20260927 23:31:09.129135 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 137329us
I20260927 23:31:09.047008 3480 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:09.134945 1554 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.129411 2293 rpcz_store.cc:276] 0927 23:31:07.805482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.805664 (+ 182us) service_pool.cc:227] Handling call
0927 23:31:07.806039 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.822157 (+ 16118us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.822213 (+ 56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.830492 (+ 8279us) log.cc:842] Serialized 26 byte log entry
0927 23:31:08.031612 (+201120us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:08.031664 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:08.031750 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:09.061042 (+1029292us) raft_consensus.cc:1671] finished
0927 23:31:09.061060 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:09.061201 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":162,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":16049,"thread_start_us":178648,"threads_started":1,"wal-append.queue_time_us":1051171}
I20260927 23:31:09.093394 3146 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:09.106962 3450 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:09.135013 1656 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:09.122232 1728 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:09.078361 3337 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:09.143136 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.167409 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:09.167757 1655 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:09.168259 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.168416 3473 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:09.169294 3458 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260927 23:31:09.169494 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.169996 3450 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.171381 3358 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.173120 3301 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.167555 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.134795 3098 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:09.161883 1627 leader_election.cc:398] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:09.147126 3471 raft_consensus.cc:686] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:09.211441 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 49636.1us
W20260927 23:31:09.214394 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 48817.3us
I20260927 23:31:09.216428 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.217142 1662 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:09.217630 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.218390 1658 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:09.156304 3361 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:09.273880 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.157230 3468 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.151589 3477 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:09.159029 3478 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:09.149420 3175 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
I20260927 23:31:09.116317 3448 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.134469 3472 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:09.281318 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 66112.9us
I20260927 23:31:09.283057 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:09.560186 1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1203) took 2315 ms (client timeout 2013 ms). Trace:
W20260927 23:31:09.567593 1737 rpcz_store.cc:269] 0927 23:31:07.245120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.542060 (+296940us) service_pool.cc:227] Handling call
0927 23:31:09.560017 (+2017957us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54002be654. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:09.560118 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1281184512}
W20260927 23:31:09.560856 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 348469us
I20260927 23:31:09.478909 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:09.561308 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:09.594542 1663 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:09.555217 1688 heartbeater.cc:644] Failed to heartbeat to 127.0.99.62:38077 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.99.62:38077 timed out after 15.000s (SENT)
W20260927 23:31:09.556089 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 397308us
I20260927 23:31:09.175312 3416 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.561510 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1538) took 2222 ms. Trace:
W20260927 23:31:09.555812 3485 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.434s user 0.001s sys 0.000s
I20260927 23:31:09.174250 3480 raft_consensus.cc:686] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:09.629874 1656 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:09.483919 3358 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
W20260927 23:31:09.350144 3239 consensus_meta.cc:287] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.777s user 0.010s sys 0.004s
I20260927 23:31:09.346762 3459 raft_consensus.cc:686] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:09.279577 1654 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:09.480031 3449 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:09.655177 3422 log.cc:924] Time spent T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 2.038s user 0.036s sys 0.031s
I20260927 23:31:09.656683 1627 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:09.408553 3126 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.328502 3436 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:09.597849 3454 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:09.621719 3214 consensus_meta.cc:287] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.781s user 0.009s sys 0.004s
I20260927 23:31:09.612779 3439 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:09.213726 3146 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.629364 3052 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.637859 3358 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:09.479336 1735 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:31:09.214015 3455 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.326241 3098 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.359s
I20260927 23:31:09.233609 3361 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.610210 1659 rpcz_store.cc:276] 0927 23:31:07.339011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.483419 (+144408us) service_pool.cc:227] Handling call
0927 23:31:07.507185 (+ 23766us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:07.507649 (+ 464us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:07.507686 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:07.508214 (+ 528us) log.cc:842] Serialized 26 byte log entry
0927 23:31:07.822531 (+314317us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:07.822566 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:07.822645 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:09.561304 (+1738659us) raft_consensus.cc:1671] finished
0927 23:31:09.561331 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:09.561452 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2010,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":313935,"threads_started":1,"wal-append.queue_time_us":1314518}
W20260927 23:31:09.702302 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 139036us
I20260927 23:31:09.280573 3461 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:09.646131 3492 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:09.555279 3484 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:09.691440 3239 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:09.618906 3478 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.768152 3484 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.702874 3279 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.327500 3466 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.374s
I20260927 23:31:09.555603 3482 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:09.790982 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 170181us
I20260927 23:31:09.791376 1656 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:09.792347 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:09.728433 3126 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:09.795910 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 182564us
I20260927 23:31:09.733503 3465 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.781379 3492 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:09.812054 1627 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.773082 3301 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.705901 3214 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.633481 3477 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.814034 1661 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:09.823169 3507 raft_consensus.cc:2741] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:31:09.387974 3453 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.676s
I20260927 23:31:09.804118 3468 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 23:31:09.820444 3252 consensus_meta.cc:287] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.852s user 0.009s sys 0.002s
I20260927 23:31:09.700577 3471 consensus_queue.cc:237] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.770346 3461 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.803875 3482 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:09.798508 3470 consensus_queue.cc:1046] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.405s
I20260927 23:31:09.807555 3480 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.889091 1663 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:09.917994 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:09.862231 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 68905.9us
I20260927 23:31:09.919989 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.876055 3448 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:09.866183 3392 consensus_meta.cc:287] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.221s user 0.002s sys 0.010s
W20260927 23:31:09.865545 3282 consensus_meta.cc:287] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.032s user 0.006s sys 0.002s
I20260927 23:31:09.905315 3472 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.880224 3455 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.906980 1588 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:09.909592 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 495084us
W20260927 23:31:09.878041 3054 consensus_meta.cc:287] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.877s user 0.003s sys 0.003s
I20260927 23:31:09.896030 3477 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.834467 3468 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:09.830685 3496 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:09.914717 3337 consensus_meta.cc:287] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.747s user 0.010s sys 0.000s
I20260927 23:31:09.886762 3146 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.896119 3052 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:09.879341 3478 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.906901 1735 raft_consensus.cc:595] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:09.960726 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:09.962175 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.892172 3450 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.872042 3484 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.967293 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:09.917165 3507 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:09.995991 2293 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:09.995368 1735 raft_consensus.cc:603] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 23:31:10.002204 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.900s user 0.014s sys 0.019s
W20260927 23:31:10.002568 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.900s user 0.014s sys 0.019s
I20260927 23:31:10.003373 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:09.816382 3481 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.174s
I20260927 23:31:10.010536 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:09.970374 3468 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:10.015452 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:10.015879 1582 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:09.798007 3239 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:10.021572 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 60946.8us
I20260927 23:31:10.024557 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:09.863368 3358 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.024724 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:09.824194 3492 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.035456 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.010413 3478 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:09.964658 3282 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:09.996923 3426 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:10.008694 1731 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.014s sys 0.034s
I20260927 23:31:09.961688 3392 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:10.032179 1551 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.992s (SENT) [suppressed 16 similar messages]
I20260927 23:31:10.052552 3361 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:10.055296 3361 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:10.055820 1553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.000s sys 0.003s
W20260927 23:31:10.056157 1553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.000s sys 0.004s
I20260927 23:31:10.057457 3361 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.044466 3358 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.058205 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.058812 1587 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:10.036756 2257 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:10.074959 3474 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.036185 1581 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:10.040386 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.092362 1663 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:10.047295 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.022432 3301 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:09.816154 3459 consensus_queue.cc:237] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.047194 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.066048 1587 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:10.039501 3098 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:10.044363 1628 connection.cc:232] RPC call timeout handler was delayed by -1.3817133009999907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
I20260927 23:31:09.971961 3337 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.044140 3478 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:10.074486 3361 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:10.047788 3143 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:10.078266 1702 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.931s user 0.009s sys 0.036s
W20260927 23:31:10.118909 1702 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.972s user 0.009s sys 0.037s
W20260927 23:31:10.119298 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 973026us
W20260927 23:31:10.105089 3452 consensus_meta.cc:287] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.957s user 0.006s sys 0.004s
I20260927 23:31:10.080601 3482 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:10.080516 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:10.081696 3239 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:09.967769 3054 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:10.078015 1728 consensus_meta.cc:287] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.793s user 0.003s sys 0.006s
I20260927 23:31:10.130641 1728 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:10.131328 1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1510) took 2779 ms (client timeout 2263 ms). Trace:
I20260927 23:31:10.131966 3483 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:10.133131 3492 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.114178 3474 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.136114 3474 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.136969 3474 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.137481 3474 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.114459 3478 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.138552 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.140188 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.141552 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:10.131773 1728 rpcz_store.cc:269] 0927 23:31:07.351836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.685431 (+333595us) service_pool.cc:227] Handling call
0927 23:31:08.975932 (+1290501us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b5800282098. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:10.131232 (+1155300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18040192}
I20260927 23:31:10.142683 3361 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.145210 3361 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.032716 3052 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.087447 3392 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:09.972323 3496 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.035552 1585 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:10.064776 3450 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.093353 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:10.090677 1554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.764s user 0.016s sys 0.047s
I20260927 23:31:10.072693 3214 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:10.063834 1731 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s user 0.014s sys 0.034s
W20260927 23:31:10.105819 1628 connection.cc:232] RPC call timeout handler was delayed by -1.2821238189999917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
I20260927 23:31:10.106232 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.106989 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.111353 3488 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:10.117344 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 214) took 1506 ms. Trace:
I20260927 23:31:10.217346 1579 rpcz_store.cc:276] 0927 23:31:08.610891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:08.611039 (+ 148us) service_pool.cc:227] Handling call
0927 23:31:08.644989 (+ 33950us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:08.953259 (+308270us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:08.953290 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:08.953329 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:08.953353 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:10.117108 (+1163755us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001ff4d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:10.117158 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:10.117288 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":708500864}
I20260927 23:31:10.118119 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1351) took 2781 ms. Trace:
I20260927 23:31:10.219789 1586 rpcz_store.cc:276] 0927 23:31:07.336933 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.483851 (+146918us) service_pool.cc:227] Handling call
0927 23:31:07.501433 (+ 17582us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:08.040326 (+538893us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:10.117884 (+2077558us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b540038f4d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:10.118042 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1076738432}
I20260927 23:31:10.122364 2293 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:10.124181 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:10.123257 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.125119 3452 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.126222 3461 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.127010 3143 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:10.186830 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 150381us
I20260927 23:31:10.150789 3052 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:10.191769 1554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.976s user 0.017s sys 0.047s
I20260927 23:31:10.213064 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.213195 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.214718 3535 raft_consensus.cc:3032] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:10.215554 3509 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:10.234284 2257 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:10.235673 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.270084 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.179145 3239 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.055s
I20260927 23:31:10.272061 1663 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:10.270083 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.274338 1586 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:10.271862 3214 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:10.275278 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.210098 3392 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:10.276993 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:10.278584 1581 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:10.281618 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:10.281957 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.275745 1583 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:10.283916 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.284024 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:10.284444 3427 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 1.736s user 0.013s sys 0.016s
I20260927 23:31:10.236346 1655 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:10.285147 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.285260 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.286242 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.287937 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:10.288897 3525 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:10.272425 3535 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:10.275142 3214 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:10.289021 3529 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260927 23:31:10.277712 3098 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:10.281594 3282 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.289675 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.291862 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:10.297225 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 1.16228e+06us
I20260927 23:31:10.290961 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:10.311630 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.317884 3352 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:10.292340 3496 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:10.290884 3516 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:10.347076 1584 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:10.347734 3536 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260927 23:31:10.231700 3488 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.349774 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.351217 1586 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:10.353854 1579 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:10.328840 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.293182 3478 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.293601 3452 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.333019 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:10.286242 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.310684 3525 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:10.292654 3524 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:10.289276 3484 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260927 23:31:10.346383 3054 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.279554 3509 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.216285 3450 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.376260 3509 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.377003 3450 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.378407 3524 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:10.379303 3524 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.380355 3524 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.379973 3468 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.381330 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.381966 1580 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:10.329070 1554 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:10.355836 3337 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:10.350657 3352 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:10.362063 3143 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.363080 3214 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.363443 3535 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.363958 3478 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.382124 3468 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.355625 1659 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:10.346781 1583 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:10.383225 1582 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:10.349128 3516 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:10.394721 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:10.393752 3054 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:10.365322 1585 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.383414 3525 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:10.401688 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.404806 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:10.405093 2291 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.396516 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:10.411593 1586 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:10.408501 3516 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:10.408643 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:10.408928 3535 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:10.405809 3352 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.412640 2294 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 2: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:10.402943 3474 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.410259 3488 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.421273 1580 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:10.413558 1659 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
W20260927 23:31:10.411542 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 82540.1us
W20260927 23:31:10.426051 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 348614us
I20260927 23:31:10.427302 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.428355 1551 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.429680 2257 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.430720 3274 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:10.431149 3274 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.431592 3274 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:10.448874 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:10.400273 3445 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.871s user 0.011s sys 0.032s
I20260927 23:31:10.412472 3054 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.541249 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.542228 2294 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:10.542780 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 164) took 1324 ms. Trace:
W20260927 23:31:10.535010 3445 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.006s user 0.011s sys 0.032s
I20260927 23:31:10.534296 3488 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:10.543910 1657 rpcz_store.cc:276] 0927 23:31:09.218827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:09.218966 (+ 139us) service_pool.cc:227] Handling call
0927 23:31:09.614614 (+395648us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:09.655428 (+ 40814us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:09.655464 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:09.656082 (+ 618us) log.cc:842] Serialized 52 byte log entry
0927 23:31:09.966981 (+310899us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:10.020479 (+ 53498us) log.cc:842] Serialized 14 byte log entry
0927 23:31:10.020781 (+ 302us) log.cc:842] Serialized 14 byte log entry
0927 23:31:10.020909 (+ 128us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:10.020973 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:10.542584 (+521611us) raft_consensus.cc:1671] finished
0927 23:31:10.542603 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:10.542725 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15375,"prepare.queue_time_us":97207,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":6434,"raft.queue_time_us":249473,"raft.run_cpu_time_us":456,"raft.run_wall_time_us":29260,"spinlock_wait_cycles":18644736,"thread_start_us":305427,"threads_started":1,"wal-append.queue_time_us":456861}
I20260927 23:31:10.538686 3474 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:10.540766 3117 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:10.563023 1627 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.022s sys 0.038s
I20260927 23:31:10.571950 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:10.573217 1627 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s user 0.023s sys 0.040s
W20260927 23:31:10.575708 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 547017us
I20260927 23:31:10.551425 3098 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:10.577116 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.578895 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:10.579276 1627 leader_election.cc:398] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:10.580967 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.583655 1627 leader_election.cc:401] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:31:10.562754 1655 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:10.583313 1657 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:10.587850 2290 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:10.588068 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.588621 1657 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:10.589067 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.589481 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.590034 1657 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:10.592906 3518 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260927 23:31:10.592479 3274 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.593299 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:10.594349 3537 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260927 23:31:10.594676 2288 raft_consensus.cc:2396] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:10.595638 3054 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:10.590210 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:10.595896 3274 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.591928 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:10.596398 1658 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:10.590386 3538 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:10.597980 1658 raft_consensus.cc:1261] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:10.602885 3452 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:10.603883 3538 raft_consensus.cc:3032] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:10.604231 3538 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:10.605115 3538 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.606434 3538 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:10.609277 1627 connection.cc:232] RPC call timeout handler was delayed by -2.2159344309999938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260927 23:31:10.609812 1627 connection.cc:232] RPC call timeout handler was delayed by -2.1241344310000159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
I20260927 23:31:10.565250 3474 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:10.558908 3488 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:10.635690 1586 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:10.637261 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:10.637295 1586 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260927 23:31:10.631847 3474 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:10.711971 1586 raft_consensus.cc:1261] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 23:31:10.712143 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75013.1us
I20260927 23:31:10.635113 3488 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:10.707625 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.723261 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.746598 2291 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:10.586908 3098 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.757997 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 216) took 2110 ms. Trace:
I20260927 23:31:10.692638 3061 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:10.759721 1662 raft_consensus.cc:1226] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:10.744411 3337 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:10.792518 1730 rpcz_store.cc:276] 0927 23:31:08.648074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:08.670748 (+ 22674us) service_pool.cc:227] Handling call
0927 23:31:08.671101 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:09.118202 (+447101us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:09.118232 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:09.118272 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:09.118298 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:10.757736 (+1639438us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b5400147154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:10.757797 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:10.757933 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1027768064}
W20260927 23:31:10.796227 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 73071.3us
I20260927 23:31:10.798233 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.797041 3525 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:10.797339 3548 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260927 23:31:10.803201 3525 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:10.803377 3361 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:10.809525 1554 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:10.815181 1551 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.809777 3459 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:10.824887 3459 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:10.803962 3361 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.828158 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:10.828276 1585 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:10.831274 2259 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:10.814105 1654 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:10.811857 3337 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:10.816768 2288 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:10.834991 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:10.842373 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.829507 3480 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:10.828604 3528 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:10.832252 3361 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:10.856355 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 62581us
I20260927 23:31:10.836616 3525 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:10.833665 3546 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:10.868603 1662 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260927 23:31:10.869311 1662 pending_rounds.cc:76] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 2
I20260927 23:31:10.869663 1662 pending_rounds.cc:97] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 23:31:10.869956 1662 raft_consensus.cc:2866] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:10.825886 3459 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.872476 2286 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:10.836119 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:10.876049 3545 log.cc:924] Time spent T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260927 23:31:10.870983 3433 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20260927 23:31:10.833617 3503 consensus_queue.cc:1035] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 23:31:10.856680 3480 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:10.835903 2906 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260927 23:31:10.857581 3528 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:10.869155 3546 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:10.854954 3214 consensus_queue.cc:1046] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260927 23:31:10.874009 3520 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:10.878598 3525 raft_consensus.cc:686] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:10.904564 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 692416us
I20260927 23:31:10.912518 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:56992 (request call id 8) took 1583 ms. Trace:
I20260927 23:31:10.930299 1735 rpcz_store.cc:276] 0927 23:31:09.354641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:09.354914 (+ 273us) service_pool.cc:227] Handling call
0927 23:31:10.912323 (+1557409us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54002eefd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a69bc3 00007f97559b381a 00007f974f0a6d04 00007f974f0a6c92 00007f974f0a6bc1 00007f974f0a6b3c 00007f974f0a5df2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:10.912427 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443060096}
I20260927 23:31:10.882036 3480 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:10.891889 3543 log.cc:924] Time spent T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
W20260927 23:31:10.935791 1628 connection.cc:232] RPC call timeout handler was delayed by -1.1089152210000179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
I20260927 23:31:10.903666 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:10.938001 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:10.938421 1658 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:10.938757 3488 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.918500 3423 raft_consensus.cc:3032] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:10.939687 3423 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:10.939903 3455 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:10.940815 3488 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.940470 3423 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:10.916520 3544 log.cc:924] Time spent T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.159s user 0.001s sys 0.001s
I20260927 23:31:10.940428 3455 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:10.894593 3546 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:10.893923 3528 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:10.914774 3337 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:10.949922 3547 log.cc:924] Time spent T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260927 23:31:10.898416 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.901481 3214 raft_consensus.cc:3032] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:10.973711 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1551) took 1151 ms. Trace:
I20260927 23:31:10.974828 1660 rpcz_store.cc:276] 0927 23:31:09.822260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.033039 (+210779us) service_pool.cc:227] Handling call
0927 23:31:10.033408 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:10.399784 (+366376us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:10.402149 (+ 2365us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:10.425535 (+ 23386us) log.cc:842] Serialized 26 byte log entry
0927 23:31:10.818449 (+392914us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:10.819321 (+ 872us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:10.819388 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:10.973510 (+154122us) raft_consensus.cc:1671] finished
0927 23:31:10.973532 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:10.973659 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":142252,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":20648,"thread_start_us":749498,"threads_started":2,"wal-append.queue_time_us":440527}
I20260927 23:31:10.980382 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:10.981034 1660 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:10.952422 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.934243 3520 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:10.970196 3098 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.971691 3072 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:10.973991 3214 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:10.996894 1628 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:10.993227 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:10.976390 1658 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:11.001488 1628 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:11.044626 3555 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260927 23:31:11.045433 3552 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260927 23:31:10.982064 3525 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:11.046038 3490 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.488s user 0.000s sys 0.001s
I20260927 23:31:11.046301 1551 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.042951 3459 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:11.061887 1660 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:11.096028 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:11.066953 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.076192 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:11.076727 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 198967us
I20260927 23:31:11.009382 2286 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:11.087802 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 86381.4us
I20260927 23:31:11.107465 1628 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:11.072593 3535 consensus_meta.cc:287] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.657s user 0.009s sys 0.000s
I20260927 23:31:11.098066 1660 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:11.111296 1628 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:11.086647 3424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.120s user 0.018s sys 0.011s
I20260927 23:31:11.114056 1628 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:11.085898 3513 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:11.106730 3516 consensus_meta.cc:287] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.688s user 0.000s sys 0.010s
I20260927 23:31:11.115554 2294 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:11.071564 3474 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.118499 3513 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.119288 3516 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.115269 3054 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.120586 3524 raft_consensus.cc:2741] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260927 23:31:11.108081 3098 consensus_queue.cc:1046] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260927 23:31:11.069412 3214 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:11.115494 3424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.220s user 0.019s sys 0.011s
I20260927 23:31:11.113234 3538 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:11.134536 3520 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.137540 1586 raft_consensus.cc:1226] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:11.139267 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:11.135294 3513 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.144080 3516 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.148721 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:11.152338 2290 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260927 23:31:11.153908 1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1527) took 3901 ms (client timeout 1992 ms). Trace:
W20260927 23:31:11.154419 1661 rpcz_store.cc:269] 0927 23:31:07.252416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.253418 (+ 1002us) service_pool.cc:227] Handling call
0927 23:31:07.421562 (+168144us) spinlock_profiling.cc:243] Waited 8.13 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:11.153682 (+3732120us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b540013bad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.153801 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5115867264}
W20260927 23:31:11.159713 3556 log.cc:924] Time spent T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260927 23:31:11.160590 3098 consensus_queue.cc:1225] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Peer 0fe98f486bab4d0b9c421c145d83f1ce log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 23:31:11.161679 3214 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.189893 3524 consensus_queue.cc:1046] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:11.190840 3524 raft_consensus.cc:3032] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:11.191139 3524 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:11.191917 3524 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.193046 3524 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.196071 3474 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.196697 1658 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260927 23:31:11.198720 3549 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.398s user 0.000s sys 0.004s
I20260927 23:31:11.193663 3546 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.160871 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:11.202514 3459 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.204242 3546 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.204591 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 217) took 1147 ms. Trace:
I20260927 23:31:11.204905 1587 rpcz_store.cc:276] 0927 23:31:10.057004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.059473 (+ 2469us) service_pool.cc:227] Handling call
0927 23:31:10.059753 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:10.101529 (+ 41776us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.204370 (+1102841us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b540038dbd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:11.204542 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14103808}
W20260927 23:31:11.205782 3540 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260927 23:31:11.207216 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1236) took 1168 ms. Trace:
I20260927 23:31:11.207311 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1559) took 1137 ms. Trace:
I20260927 23:31:11.207665 1663 rpcz_store.cc:276] 0927 23:31:10.069958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.092947 (+ 22989us) service_pool.cc:227] Handling call
0927 23:31:11.207176 (+1114229us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b54003af7d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.207268 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136627456}
I20260927 23:31:11.208765 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1355) took 1125 ms. Trace:
I20260927 23:31:11.209082 1579 rpcz_store.cc:276] 0927 23:31:10.083351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.287306 (+203955us) service_pool.cc:227] Handling call
0927 23:31:11.208632 (+921326us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b5400467654. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.208718 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61015808}
I20260927 23:31:11.212013 1659 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:11.207628 1584 rpcz_store.cc:276] 0927 23:31:10.038886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.047099 (+ 8213us) service_pool.cc:227] Handling call
0927 23:31:11.207075 (+1159976us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400466754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.207169 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":583732608}
W20260927 23:31:11.219527 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 173285us
I20260927 23:31:11.221678 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:11.217170 1660 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:11.223666 1551 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.225677 3513 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:11.220548 3455 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.229308 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.227474 3535 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:11.232761 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 219) took 1175 ms. Trace:
I20260927 23:31:11.243434 3467 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:11.244486 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 1.66249e+06us
I20260927 23:31:11.250150 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:11.233291 1583 rpcz_store.cc:276] 0927 23:31:10.057541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.407121 (+349580us) service_pool.cc:227] Handling call
0927 23:31:11.144620 (+737499us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:11.144804 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:11.146734 (+ 1930us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:11.146798 (+ 64us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:11.146835 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.146895 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:11.232520 (+ 85625us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540007dd50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.232688 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1695347584}
W20260927 23:31:11.255996 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 134635us
W20260927 23:31:11.258690 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 182560us
I20260927 23:31:11.259545 1554 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:11.260569 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.264135 1628 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.241356 3516 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:11.267158 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1359) took 1182 ms (client timeout 1548 ms). Trace:
I20260927 23:31:11.251585 3539 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:11.273135 3459 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260927 23:31:11.273978 3459 raft_consensus.cc:3032] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:11.274309 3459 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:11.267661 1581 rpcz_store.cc:269] 0927 23:31:10.084309 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.100262 (+ 15953us) service_pool.cc:227] Handling call
0927 23:31:11.267009 (+1166747us) spinlock_profiling.cc:243] Waited 66.4 ms on lock 0x7b54003b47d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:11.267096 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":152703872}
I20260927 23:31:11.270006 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.271210 3470 raft_consensus.cc:3032] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 23:31:11.259862 3454 consensus_queue.cc:1046] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260927 23:31:11.271862 3502 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:11.269179 3416 consensus_queue.cc:1046] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260927 23:31:11.253010 3480 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:11.277036 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 53444.8us
I20260927 23:31:11.288512 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 206) took 2769 ms. Trace:
I20260927 23:31:11.288877 1588 rpcz_store.cc:276] 0927 23:31:08.518951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:08.519085 (+ 134us) service_pool.cc:227] Handling call
0927 23:31:08.519430 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:09.964151 (+1444721us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.288309 (+1324158us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400242a54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:11.288461 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3190651648}
I20260927 23:31:11.290292 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 355) took 1287 ms. Trace:
I20260927 23:31:11.290797 2293 rpcz_store.cc:276] 0927 23:31:10.002924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.003070 (+ 146us) service_pool.cc:227] Handling call
0927 23:31:10.003320 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:10.221060 (+217740us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.290049 (+1068989us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b540036d3d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:11.290227 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273293184}
I20260927 23:31:11.269516 1628 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.279664 3470 raft_consensus.cc:729] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:11.275111 3516 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.265357 3393 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:11.241089 3361 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.284458 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.255206 3467 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.287276 3459 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:11.323369 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 53906us
I20260927 23:31:11.330245 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.325588 1554 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:11.286821 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1549) took 1488 ms. Trace:
I20260927 23:31:11.328370 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:11.333266 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.289047 3488 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:11.286064 3535 raft_consensus.cc:686] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:11.334863 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 44467.6us
I20260927 23:31:11.331250 1656 rpcz_store.cc:276] 0927 23:31:09.798774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:09.831255 (+ 32481us) service_pool.cc:227] Handling call
0927 23:31:09.869209 (+ 37954us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:09.869425 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:09.869453 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:11.126089 (+1256636us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b2c0005b518. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:11.126472 (+ 383us) log.cc:842] Serialized 26 byte log entry
0927 23:31:11.251296 (+124824us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:11.251346 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.251522 (+ 176us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:11.282798 (+ 31276us) raft_consensus.cc:1671] finished
0927 23:31:11.286495 (+ 3697us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:11.286750 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73367424,"thread_start_us":124495,"threads_started":1,"wal-append.queue_time_us":154194}
I20260927 23:31:11.335996 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.320994 2901 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:11.285262 3502 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.332918 3520 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260927 23:31:11.337361 3356 consensus_queue.cc:1046] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:11.340991 3567 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.276185 3525 consensus_queue.cc:1046] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260927 23:31:11.345293 2289 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260927 23:31:11.324848 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.346451 3459 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.348119 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.349289 3516 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.350453 3480 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.284876 3528 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.356756 3461 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:11.377281 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 128134us
W20260927 23:31:11.417091 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 142432us
I20260927 23:31:11.417239 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:11.378902 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 48718.2us
I20260927 23:31:11.418879 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.379374 3117 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:11.381430 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.422037 2294 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
W20260927 23:31:11.380414 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 44494.6us
I20260927 23:31:11.381733 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:11.383363 3500 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:11.329432 3524 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:11.357118 3467 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.387539 1554 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.357128 3393 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:11.352841 3436 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.390647 3445 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:11.399775 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.379993 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.390796 3416 raft_consensus.cc:3032] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:11.357252 3252 raft_consensus.cc:3032] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:11.379747 3453 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.406134 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:11.390652 3488 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:11.334626 3503 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.059s
I20260927 23:31:11.376824 3457 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:11.423236 1588 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:11.434157 1628 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.451524 2259 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:11.423412 1703 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:11.441138 2901 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:11.453128 3497 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.453317 2901 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:11.421756 3481 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.422091 3496 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:11.426664 3455 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.452834 3502 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.441109 1700 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:11.437239 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 49734.9us
I20260927 23:31:11.429819 3524 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:11.489207 3524 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:11.489292 3279 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.448589 3252 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:11.430596 3117 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:11.487751 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 46724.6us
I20260927 23:31:11.483903 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:11.457213 1628 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.475383 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:11.435154 3535 consensus_queue.cc:237] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.490939 1554 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:11.438479 3500 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:11.431777 3480 consensus_queue.cc:1046] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260927 23:31:11.473934 3439 consensus_queue.cc:1046] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260927 23:31:11.439510 2288 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:11.445364 3380 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.449060 1728 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:11.500372 1700 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:11.479511 3528 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.445609 3416 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:11.456244 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:11.501571 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.442294 3461 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.501894 1703 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.472165 3468 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:11.501016 3279 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260927 23:31:11.539422 3279 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.539234 3520 consensus_queue.cc:1046] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:11.540185 3279 raft_consensus.cc:3032] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:11.540611 3279 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:11.514854 3480 consensus_queue.cc:1225] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260927 23:31:11.542856 3252 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.543946 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:11.454311 3453 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.545418 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.546546 3453 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.506711 1735 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:11.552460 3571 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.210s user 0.003s sys 0.001s
I20260927 23:31:11.483510 3361 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:11.563593 1579 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 23:31:11.563937 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 62141.5us
I20260927 23:31:11.503189 3467 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20260927 23:31:11.509822 3439 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:11.476562 3525 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:11.438819 3445 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:11.445861 3471 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:11.517788 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 61658.7us
I20260927 23:31:11.575896 1581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:11.441864 3436 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.492156 3481 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.514602 3380 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.492311 3457 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.524698 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.526077 3461 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.526582 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:11.527247 1702 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.530685 2257 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.531572 3567 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.568962 3439 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:11.605163 3457 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.533473 3450 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:11.535779 3575 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.503269 3491 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:11.503672 3500 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.491947 3496 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.510632 3455 raft_consensus.cc:2741] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:31:11.554875 3502 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:11.556257 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:11.488991 2901 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.615859 1655 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:11.507311 3503 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:11.507879 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 50766.3us
I20260927 23:31:11.567723 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:11.634897 3436 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.635118 1656 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:11.636161 3464 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:11.581815 3492 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260927 23:31:11.640726 1579 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:11.641070 1588 raft_consensus.cc:1226] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:11.519413 3448 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:11.472056 3497 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.573751 1735 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:11.546216 1728 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:11.478803 3488 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.563480 3454 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.541599 3279 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.594698 1730 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:11.595023 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 64455.9us
W20260927 23:31:11.595055 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 67933.7us
I20260927 23:31:11.595969 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:11.576931 3445 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.570467 3525 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:11.531774 3416 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:11.598690 1581 raft_consensus.cc:595] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:11.593953 3465 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:11.599562 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.600234 3337 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:11.599601 3567 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.595026 3471 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260927 23:31:11.599887 3380 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.601315 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:11.602785 2287 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:11.603022 3117 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:11.603231 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.603523 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:11.604728 3481 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:11.570921 3516 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260927 23:31:11.608378 3491 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:11.597352 3449 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:11.645421 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:11.905545 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 326867us
W20260927 23:31:11.974659 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 418467us
I20260927 23:31:11.981182 1628 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:11.975955 1662 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:11.978528 1661 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:11.983033 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:11.881671 3581 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
W20260927 23:31:11.922138 3506 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s user 0.014s sys 0.047s
I20260927 23:31:11.700544 3575 consensus_queue.cc:1046] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.357s
I20260927 23:31:12.061673 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1585) took 1128 ms. Trace:
I20260927 23:31:12.062379 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:12.065644 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 82699.5us
I20260927 23:31:11.710373 3356 consensus_queue.cc:1046] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.356s
W20260927 23:31:11.901479 3576 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.522s user 0.001s sys 0.000s
I20260927 23:31:12.067569 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:11.631644 3455 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:11.899875 3274 raft_consensus.cc:3032] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:12.068358 1627 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:11.715024 3450 consensus_queue.cc:1225] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 23:31:11.984906 3448 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:11.938277 3473 raft_consensus.cc:3032] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:12.096367 1663 rpcz_store.cc:276] 0927 23:31:10.932987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.208535 (+275548us) service_pool.cc:227] Handling call
0927 23:31:11.208840 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:11.209044 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:11.209066 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:11.209490 (+ 424us) log.cc:842] Serialized 26 byte log entry
0927 23:31:11.255605 (+ 46115us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:11.255642 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.255715 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:11.551904 (+296189us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:11.629115 (+ 77211us) raft_consensus.cc:1671] finished
0927 23:31:11.629134 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:12.061571 (+432437us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25663232,"thread_start_us":45796,"threads_started":1,"wal-append.queue_time_us":132398}
I20260927 23:31:11.986824 3491 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:11.973474 3524 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.956013 3470 consensus_queue.cc:260] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.622908 3439 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:11.992597 3539 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.094775 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:11.937728 3393 raft_consensus.cc:3032] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
W20260927 23:31:12.141623 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 46982.9us
I20260927 23:31:11.986696 3525 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:11.992957 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 213) took 3361 ms. Trace:
I20260927 23:31:11.987427 3252 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.096140 1730 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:12.114073 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1509) took 4763 ms. Trace:
I20260927 23:31:12.008200 1732 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:12.067086 3117 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:12.144431 3583 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260927 23:31:12.154134 1733 rpcz_store.cc:276] 0927 23:31:07.351124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:07.860248 (+509124us) service_pool.cc:227] Handling call
0927 23:31:10.042622 (+2182374us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b5800282098. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:10.042846 (+ 224us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:11.272011 (+1229165us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:11.272050 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:11.272473 (+ 423us) log.cc:842] Serialized 26 byte log entry
0927 23:31:11.327976 (+ 55503us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:11.328021 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.328868 (+ 847us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:11.900775 (+571907us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:12.113840 (+213065us) raft_consensus.cc:1671] finished
0927 23:31:12.113862 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:12.114001 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1228780,"raft.queue_time_us":1229791,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":11912320,"thread_start_us":55155,"threads_started":1,"wal-append.queue_time_us":104306}
I20260927 23:31:12.082593 3457 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:12.008653 1654 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:12.103011 3274 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:12.088565 3497 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.207670 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.112087 3455 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:12.143419 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:12.220937 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 77639.3us
I20260927 23:31:12.104022 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:12.086848 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:11.811898 2901 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.192829 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.140980 3393 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:12.194120 3470 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:12.110244 3448 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.057858 3465 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.199987 1662 raft_consensus.cc:2396] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 23:31:12.102764 3416 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.082731 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:12.113440 3423 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.205652 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:12.145110 3380 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:12.222576 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:11.629036 3496 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.222863 3455 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.262615 3497 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:12.263145 3497 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:12.263592 3497 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.223443 1627 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:12.194348 1657 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:12.196539 3117 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:12.266440 3117 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.156379 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.245019 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.113267 3500 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.251742 1729 raft_consensus.cc:2362] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:12.204421 3457 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:12.219965 3439 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:12.110536 3473 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:12.242062 3472 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:12.267192 3455 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:12.147230 3539 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.269881 1733 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:12.107401 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:12.253794 3380 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:12.283604 3457 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.284927 3524 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.285069 3457 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:12.285526 3524 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:12.282709 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:12.278692 3459 consensus_meta.cc:287] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.868s user 0.014s sys 0.000s
I20260927 23:31:12.284176 3380 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.267325 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.283939 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.276607 2290 raft_consensus.cc:2379] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:12.286602 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.285897 3524 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.296842 3459 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:12.297384 3584 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260927 23:31:12.300769 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.277164 3274 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.301340 1657 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:12.280553 3472 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:12.313971 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.321413 1733 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.305720 3524 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.323582 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:12.327565 2901 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.315230 3459 raft_consensus.cc:2741] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260927 23:31:12.316272 3472 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.333627 1732 raft_consensus.cc:2379] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.322613 3274 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.343751 1662 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:12.343142 3459 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.321698 3567 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.357761 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:12.363034 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 381941us
I20260927 23:31:12.369890 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.370440 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.314863 3473 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.371433 1579 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:12.352140 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.287071 3393 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.353204 3337 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:12.354825 3505 raft_consensus.cc:3032] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:12.379604 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.379834 1662 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 23:31:12.379832 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.379918 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.380532 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.346432 3468 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260927 23:31:12.382134 1658 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 23:31:12.393635 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.351817 3459 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.395452 3337 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.396551 3459 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.372268 3465 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.398483 3539 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.348485 3492 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260927 23:31:12.391929 1660 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:12.401120 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:12.398810 3585 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260927 23:31:12.373059 3500 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.428812 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.391507 3505 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:12.429747 1661 raft_consensus.cc:2379] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:12.431334 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.435598 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.435905 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.436246 1661 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:12.436830 1663 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:12.393452 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:12.402586 1583 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:12.399329 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:12.442028 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.404904 3393 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.425333 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.148829 1731 rpcz_store.cc:276] 0927 23:31:08.631144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:08.633853 (+ 2709us) service_pool.cc:227] Handling call
0927 23:31:08.664941 (+ 31088us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:08.782508 (+117567us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:09.047284 (+264776us) log.cc:842] Serialized 14 byte log entry
0927 23:31:11.276926 (+2229642us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:11.279179 (+ 2253us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:11.283786 (+ 4607us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.957730 (+673944us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b18003f94c0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db14f96 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9
0927 23:31:11.968583 (+ 10853us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b5400105754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:11.968667 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:11.989096 (+ 20429us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1136334,"prepare.queue_time_us":303952,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":1056833,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":8021,"spinlock_wait_cycles":156373504,"thread_start_us":1252774,"threads_started":2,"wal-append.queue_time_us":1161482,"wal-append.run_cpu_time_us":1110,"wal-append.run_wall_time_us":1001136}
W20260927 23:31:12.475440 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 165290us
I20260927 23:31:12.477268 1661 raft_consensus.cc:1226] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 2. Ops: []
I20260927 23:31:12.453270 1655 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260927 23:31:12.481531 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 118724us
I20260927 23:31:12.487154 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:12.487800 1661 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.488883 1732 raft_consensus.cc:2379] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.481860 2285 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260927 23:31:12.500072 1700 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38820 dropped due to backpressure. The service queue is full; it has 50 items.
I20260927 23:31:12.481495 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:12.496172 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 131130us
I20260927 23:31:12.452725 3590 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260927 23:31:12.516603 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:12.517148 1661 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:12.518003 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:12.494693 3450 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.488775 3594 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.524009 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.476235 3445 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.494903 3535 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.521768 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 403) took 1254 ms. Trace:
I20260927 23:31:12.527853 2291 rpcz_store.cc:276] 0927 23:31:11.267294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.277927 (+ 10633us) service_pool.cc:227] Handling call
0927 23:31:11.278341 (+ 414us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:11.504040 (+225699us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:11.504068 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:11.510130 (+ 6062us) log.cc:842] Serialized 26 byte log entry
0927 23:31:11.597892 (+ 87762us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:11.597930 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.597988 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:12.521582 (+923594us) raft_consensus.cc:1671] finished
0927 23:31:12.521604 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:12.521716 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15812,"raft.queue_time_us":9692,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":29875,"thread_start_us":287085,"threads_started":2,"wal-append.queue_time_us":716821}
I20260927 23:31:12.496549 3510 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.559878 3510 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.561724 3510 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.561887 1659 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:12.562354 1659 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:12.526161 3450 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.563390 1659 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.564703 3450 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.564839 1659 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:12.565557 3570 log.cc:924] Time spent T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.233s user 0.002s sys 0.000s
I20260927 23:31:12.566447 3491 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.536870 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.567560 1733 raft_consensus.cc:2354] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:12.568310 3491 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.568466 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.569098 1654 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:12.540257 3117 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.570653 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.571231 3117 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.571949 1729 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.522822 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.492275 3593 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260927 23:31:12.548823 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:12.521268 3589 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.553679 3472 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.557194 3455 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.558369 3445 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.558776 3535 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.484074 3505 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.527451 3356 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:12.569267 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.574168 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.636782 1661 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:12.575054 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:12.520928 3488 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260927 23:31:12.569334 1737 raft_consensus.cc:2396] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260927 23:31:12.575950 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 52028.3us
I20260927 23:31:12.571233 1656 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:12.541733 3497 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.643486 3356 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:12.647855 1659 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:12.648638 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 182) took 1074 ms. Trace:
I20260927 23:31:12.649202 1659 rpcz_store.cc:276] 0927 23:31:11.574532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.574713 (+ 181us) service_pool.cc:227] Handling call
0927 23:31:11.645986 (+ 71273us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:11.646150 (+ 164us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:12.561775 (+915625us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b5400488120. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f97509ff730 00007f9750a71300 00007f9750a72371 00007f9750a6eedf 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc
0927 23:31:12.648347 (+ 86572us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:12.648577 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1607782016}
I20260927 23:31:12.651211 1659 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:12.520196 1700 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1545)
I20260927 23:31:12.653604 1659 pending_rounds.cc:76] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 1
I20260927 23:31:12.653940 1659 pending_rounds.cc:97] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260927 23:31:12.655148 1659 raft_consensus.cc:2866] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:12.577442 3341 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.658202 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.658947 1661 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:12.656769 1700 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.661131 1700 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.631387 1655 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:12.672585 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 154038us
I20260927 23:31:12.673338 1700 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.674475 1700 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.675060 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.675398 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.676472 1700 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.677685 1700 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:12.655181 3356 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.680146 1662 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260927 23:31:12.553385 3525 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.576327 3380 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.646390 3134 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.655401 3505 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.530388 3594 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.632359 3524 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.702129 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:12.659220 3491 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.670902 3439 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:12.671437 3470 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.707942 1656 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:12.639771 3448 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.682582 1728 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.678889 3465 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:12.676836 3252 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.632705 3589 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.681632 3496 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:12.682047 3457 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.710216 1731 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:12.723129 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 45515.2us
I20260927 23:31:12.724295 1700 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:12.725646 1656 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:12.707744 3492 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.725883 1700 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:12.707623 3134 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.718026 3439 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:12.731774 3525 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.700903 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:12.733470 1730 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:12.741969 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1239) took 1223 ms. Trace:
I20260927 23:31:12.744772 1737 rpcz_store.cc:276] 0927 23:31:11.518202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.115660 (+597458us) service_pool.cc:227] Handling call
0927 23:31:12.732582 (+616922us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54003c3554. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:12.741908 (+ 9326us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688338176}
I20260927 23:31:12.742398 1700 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:12.747227 1700 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:12.748668 1700 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:12.743670 3496 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:12.762794 3496 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:12.763139 3496 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:12.738111 3380 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.800215 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.736477 3457 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:12.718021 3488 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260927 23:31:12.796244 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:12.804533 1588 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:12.743018 3591 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.718214 3491 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.806193 2259 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:12.810931 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:12.746918 3439 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.758124 3465 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:12.817543 1579 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:12.820714 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:12.825924 2258 leader_election.cc:334] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731): Timed out: RequestConsensusVote RPC to 127.0.99.3:35731 timed out after 1.708s (SENT)
I20260927 23:31:12.826488 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:12.744092 2290 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260927 23:31:12.747227 3484 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.751413 3252 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260927 23:31:12.807471 3457 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:12.742677 3492 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.820684 3591 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.764273 1654 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:12.809991 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:12.837054 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.837515 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:12.759984 3470 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:12.871069 1737 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:12.836796 1579 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:12.833673 3134 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.828652 3472 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.799507 3605 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.873651 3457 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:12.883529 3491 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.884975 3448 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.828420 3439 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:12.828971 3465 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:12.867266 3601 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260927 23:31:12.887311 3465 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:12.820515 3380 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260927 23:31:12.890578 3472 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:12.890914 3472 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:12.891681 3472 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:12.866597 3600 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
W20260927 23:31:12.868134 3604 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260927 23:31:12.879729 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:12.868523 3599 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
I20260927 23:31:12.868269 3455 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.871469 3470 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:12.871560 3593 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.896291 1737 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:12.877099 2259 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:12.910418 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1525) took 1499 ms. Trace:
W20260927 23:31:12.910738 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 160806us
I20260927 23:31:12.911878 1700 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:12.911007 1737 rpcz_store.cc:276] 0927 23:31:11.410593 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.879396 (+1468803us) service_pool.cc:227] Handling call
0927 23:31:12.910365 (+ 30969us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:12.870023 3252 consensus_queue.cc:1225] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 23:31:12.913625 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:12.834426 3341 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:12.911571 3470 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:12.831053 3525 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:12.920941 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 43988.4us
I20260927 23:31:12.872368 3497 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:12.839358 3535 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:12.912719 3448 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:12.897746 3500 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:12.877854 3588 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:12.842674 3484 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:12.998715 3588 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:12.911868 3465 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.000747 3588 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.001618 3465 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.001031 2289 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:13.002493 3484 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.000063 1700 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1264)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1540)
I20260927 23:31:13.003274 3356 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.005301 3356 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:13.005672 3598 log.cc:924] Time spent T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
I20260927 23:31:13.006006 2291 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:12.898180 3450 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20260927 23:31:13.006676 3450 consensus_queue.cc:1225] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 23:31:13.003681 1700 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1264)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 236)
W20260927 23:31:13.008076 1657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1592) took 1421 ms (client timeout 1766 ms). Trace:
W20260927 23:31:13.008478 1657 rpcz_store.cc:269] 0927 23:31:11.586707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.594282 (+ 7575us) service_pool.cc:227] Handling call
0927 23:31:13.007917 (+1413635us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540013a1d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.008021 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":481289088}
I20260927 23:31:13.010681 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.010865 3589 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.012586 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:12.908906 3591 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:12.994889 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 284988us
I20260927 23:31:12.960852 3252 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.019742 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:12.991470 3134 raft_consensus.cc:2781] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:12.956624 3525 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.024387 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.027460 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.031432 1662 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.016950 3448 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.032141 3448 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:12.529948 1701 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1538)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1525)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 236)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1540)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1545)
I20260927 23:31:12.998364 3500 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.039669 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.040134 3500 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.040534 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 230) took 1701 ms. Trace:
I20260927 23:31:13.042433 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:13.042887 1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 229) took 1704 ms (client timeout 1708 ms). Trace:
I20260927 23:31:13.043576 3535 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:13.025543 3594 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:12.912807 3593 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.046452 3492 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.047946 3525 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.040874 1732 rpcz_store.cc:276] 0927 23:31:11.339038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.039368 (+1700330us) service_pool.cc:227] Handling call
0927 23:31:13.040482 (+ 1114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.043059 1654 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:13.043519 1735 rpcz_store.cc:269] 0927 23:31:11.338780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.378942 (+ 40162us) service_pool.cc:227] Handling call
0927 23:31:13.042659 (+1663717us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b5400105254. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.042815 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139041536}
I20260927 23:31:13.050993 1657 raft_consensus.cc:2396] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:13.046144 3593 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.056535 3134 raft_consensus.cc:686] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:13.060678 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1246) took 1531 ms. Trace:
I20260927 23:31:13.050876 3535 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.057780 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:13.058120 3279 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:13.058909 1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1231) took 1543 ms (client timeout 2037 ms). Trace:
I20260927 23:31:13.061069 1733 rpcz_store.cc:276] 0927 23:31:11.528839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.323310 (+794471us) service_pool.cc:227] Handling call
0927 23:31:13.060516 (+737206us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54002134d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.060626 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":538184960}
W20260927 23:31:13.064864 1729 rpcz_store.cc:269] 0927 23:31:11.515235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.274014 (+758779us) service_pool.cc:227] Handling call
0927 23:31:13.058746 (+784732us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b540011bf54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.058860 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":625140480}
I20260927 23:31:13.030057 3252 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.063444 3535 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:13.015241 1703 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:60802 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1552)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1554)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1256)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1259)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1258)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1263)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1260)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1261)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1257)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1262)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1265)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1264)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254)
I20260927 23:31:13.074448 1662 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:13.044129 3457 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.041613 3448 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.067061 1583 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:13.071060 3134 consensus_queue.cc:237] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.101182 2259 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:13.102536 3523 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:13.102958 3523 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.103292 3523 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.107518 3252 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:13.109118 3586 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.033s user 0.001s sys 0.001s
I20260927 23:31:13.111152 3523 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.112071 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 220) took 1786 ms. Trace:
I20260927 23:31:13.112448 1734 rpcz_store.cc:276] 0927 23:31:11.326047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.326389 (+ 342us) service_pool.cc:227] Handling call
0927 23:31:11.326706 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:11.386817 (+ 60111us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:11.386850 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:11.390849 (+ 3999us) log.cc:842] Serialized 26 byte log entry
0927 23:31:11.699097 (+308248us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:11.699153 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:11.699251 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:13.108604 (+1409353us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:13.111767 (+ 3163us) raft_consensus.cc:1671] finished
0927 23:31:13.111788 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.112011 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59782,"raft.queue_time_us":61329,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":10601,"spinlock_wait_cycles":64831744,"thread_start_us":265702,"threads_started":1,"wal-append.queue_time_us":564257}
I20260927 23:31:13.117574 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.118165 1734 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:13.098035 3492 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:13.248252 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 94861.5us
I20260927 23:31:13.224365 3484 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:13.229171 1655 consensus_meta.cc:287] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.538s user 0.007s sys 0.003s
I20260927 23:31:13.213409 3500 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:13.107111 1654 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
W20260927 23:31:13.261224 3617 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
I20260927 23:31:13.263737 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.264994 3503 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:13.267191 3496 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:13.271874 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 69503.4us
I20260927 23:31:13.251868 3591 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.274425 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:13.278795 1584 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.282835 2294 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260927 23:31:13.265053 3523 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.239912 3439 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.271257 3484 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.285389 3591 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.302717 3439 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.303016 3484 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.285131 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:13.250280 3492 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.269670 1655 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.276542 3503 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:13.277037 1732 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.303426 2288 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260927 23:31:13.310657 3591 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.312345 3448 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.314015 3535 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.319101 1736 raft_consensus.cc:2396] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:13.286262 3457 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.272091 3500 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.322925 1584 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:13.330674 2259 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:13.333360 3500 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.333470 3503 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.334263 3492 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.334657 3503 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:13.335414 3535 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.338999 3470 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.341109 3470 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.342199 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:13.327801 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:13.347173 1583 raft_consensus.cc:2396] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:13.348461 3492 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.348959 3492 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.351207 3496 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.352301 3492 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.332891 3484 raft_consensus.cc:2781] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:13.322961 3525 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.354832 3484 raft_consensus.cc:686] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:13.355650 3484 consensus_queue.cc:237] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.356896 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:13.357511 1583 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.360626 1734 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:13.362602 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.363340 1734 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.363911 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1247) took 1834 ms. Trace:
I20260927 23:31:13.364269 1734 rpcz_store.cc:276] 0927 23:31:11.529082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.362350 (+1833268us) service_pool.cc:227] Handling call
0927 23:31:13.363863 (+ 1513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.365172 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.366156 1734 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:13.368323 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.369385 3525 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.370019 3503 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.370767 1734 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.370730 3503 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.373700 3503 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.374815 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.379446 1734 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.380257 1583 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:13.382085 2259 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:13.387800 3502 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:13.386546 3525 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:13.389757 3502 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.389973 3525 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:13.390961 1734 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.391631 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1541) took 1143 ms. Trace:
I20260927 23:31:13.390969 3525 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.392093 1734 rpcz_store.cc:276] 0927 23:31:12.248244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.374583 (+1126339us) service_pool.cc:227] Handling call
0927 23:31:13.391578 (+ 16995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.393328 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.393898 1734 raft_consensus.cc:2362] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3. Re-sending same reply.
I20260927 23:31:13.394430 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1539) took 1251 ms. Trace:
I20260927 23:31:13.394771 1734 rpcz_store.cc:276] 0927 23:31:12.143395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.393079 (+1249684us) service_pool.cc:227] Handling call
0927 23:31:13.394390 (+ 1311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.395929 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.396608 1734 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.397305 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.398006 1734 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.398800 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.400063 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.387499 1732 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.400846 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1238) took 1883 ms. Trace:
I20260927 23:31:13.401316 1734 rpcz_store.cc:276] 0927 23:31:11.517399 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.399851 (+1882452us) service_pool.cc:227] Handling call
0927 23:31:13.400780 (+ 929us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.401721 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.403241 1732 raft_consensus.cc:2396] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:13.402195 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.405848 1734 raft_consensus.cc:2379] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.405887 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.406404 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1235) took 1890 ms. Trace:
I20260927 23:31:13.424479 1734 rpcz_store.cc:276] 0927 23:31:11.516325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.401971 (+1885646us) service_pool.cc:227] Handling call
0927 23:31:13.406360 (+ 4389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.425799 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.426807 1734 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.427634 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.428290 1734 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:13.429124 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.429524 1700 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1254) took 1443 ms. Trace:
I20260927 23:31:13.430060 1734 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.431216 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.432956 1734 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:13.434134 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.435748 1734 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.436463 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.436807 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.437469 1733 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:13.437952 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1547) took 1090 ms. Trace:
I20260927 23:31:13.423331 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.416694 3524 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.437093 1734 raft_consensus.cc:2354] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:13.487543 2287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 449) took 1132 ms. Trace:
I20260927 23:31:13.450719 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 14) took 1915 ms. Trace:
I20260927 23:31:13.444768 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.438271 1733 rpcz_store.cc:276] 0927 23:31:12.348060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.436527 (+1088467us) service_pool.cc:227] Handling call
0927 23:31:13.437904 (+ 1377us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.494825 1581 rpcz_store.cc:276] 0927 23:31:11.535395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.563639 (+ 28244us) service_pool.cc:227] Handling call
0927 23:31:13.450529 (+1886890us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400060c54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a69bc3 00007f97559b381a 00007f974f0a6d04 00007f974f0a6c92 00007f974f0a6bc1 00007f974f0a6b3c 00007f974f0a5df2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.450641 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51102080}
I20260927 23:31:13.495600 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.448220 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:13.461496 1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1245) took 1932 ms (client timeout 1704 ms). Trace:
I20260927 23:31:13.496959 1658 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:13.495820 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.498377 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1448) took 1096 ms. Trace:
I20260927 23:31:13.498536 1659 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:13.499092 1736 rpcz_store.cc:269] 0927 23:31:11.528582 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.987021 (+458439us) service_pool.cc:227] Handling call
0927 23:31:13.461292 (+1474271us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540011c954. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.461430 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2812726656}
I20260927 23:31:13.445139 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.446962 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.501138 1662 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.502236 1662 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:13.499627 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:13.494777 2287 rpcz_store.cc:276] 0927 23:31:12.354657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.355233 (+ 576us) service_pool.cc:227] Handling call
0927 23:31:13.486656 (+1131423us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:13.486908 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.487141 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:13.487180 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:13.487236 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:13.487279 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.487350 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.487478 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10403584}
I20260927 23:31:13.502171 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.503759 1731 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.505084 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.505506 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.506232 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.506870 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.507318 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.508069 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.478650 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.514433 3472 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.417615 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:13.515232 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.499006 1735 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.515909 1731 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.516884 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1551) took 1085 ms. Trace:
I20260927 23:31:13.517338 1731 rpcz_store.cc:276] 0927 23:31:12.431178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.501910 (+1070732us) service_pool.cc:227] Handling call
0927 23:31:13.516816 (+ 14906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.438087 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.515926 1736 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.519274 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1543) took 1239 ms. Trace:
I20260927 23:31:13.520818 1732 rpcz_store.cc:276] 0927 23:31:12.280437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.437830 (+1157393us) service_pool.cc:227] Handling call
0927 23:31:13.519218 (+ 81388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.522008 1732 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:13.429932 1700 rpcz_store.cc:276] 0927 23:31:11.986297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.429476 (+1443179us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 23:31:13.522663 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 224) took 2185 ms. Trace:
I20260927 23:31:13.522982 1737 raft_consensus.cc:2396] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:13.523500 2291 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:13.486071 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.525333 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1548) took 1156 ms. Trace:
I20260927 23:31:13.525377 1654 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.525743 1737 rpcz_store.cc:276] 0927 23:31:12.368467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.913375 (+544908us) service_pool.cc:227] Handling call
0927 23:31:13.525164 (+611789us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b54002305d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.525288 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1398394112}
I20260927 23:31:13.527154 1737 raft_consensus.cc:1226] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:13.527724 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 225) took 2190 ms. Trace:
I20260927 23:31:13.528223 1737 rpcz_store.cc:276] 0927 23:31:11.337660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.526698 (+2189038us) service_pool.cc:227] Handling call
0927 23:31:13.526960 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.527526 (+ 566us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.527674 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.514456 1734 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:13.523018 1700 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:38812 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1555)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 240)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 242)
I20260927 23:31:13.530068 1732 rpcz_store.cc:276] 0927 23:31:11.337403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.521547 (+2184144us) service_pool.cc:227] Handling call
0927 23:31:13.521836 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.522448 (+ 612us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.522612 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.444645 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.529548 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1553) took 1033 ms. Trace:
I20260927 23:31:13.532615 1736 rpcz_store.cc:276] 0927 23:31:12.487423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.515005 (+1027582us) service_pool.cc:227] Handling call
0927 23:31:13.520515 (+ 5510us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:13.465708 3464 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260927 23:31:13.532370 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.536499 1656 raft_consensus.cc:2396] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:13.536755 1732 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:13.538604 1732 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:13.538475 3592 consensus_queue.cc:1046] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:13.539230 2285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 445) took 2465 ms. Trace:
I20260927 23:31:13.536506 1736 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:13.539547 3274 consensus_queue.cc:1046] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:13.540020 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 231) took 2200 ms. Trace:
I20260927 23:31:13.540540 3592 raft_consensus.cc:3032] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:13.540917 3592 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:13.539583 2285 rpcz_store.cc:276] 0927 23:31:11.074078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:11.079489 (+ 5411us) service_pool.cc:227] Handling call
0927 23:31:11.115092 (+ 35603us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:12.499946 (+1384854us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.538973 (+1039027us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400640ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:13.539152 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3068863360}
I20260927 23:31:13.541734 3592 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.543002 3274 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.543022 3592 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.543169 3513 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.543535 3274 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.543866 3513 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.538169 1659 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:13.545849 3613 consensus_queue.cc:1046] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 23:31:13.546918 3613 consensus_queue.cc:1225] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260927 23:31:13.547344 3470 raft_consensus.cc:3032] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:13.545636 3274 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.547865 3470 raft_consensus.cc:729] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:13.546087 3513 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.548770 3470 consensus_queue.cc:260] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.549975 3470 raft_consensus.cc:3037] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.551290 1736 rpcz_store.cc:276] 0927 23:31:11.339283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.533712 (+2194429us) service_pool.cc:227] Handling call
0927 23:31:13.536338 (+ 2626us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.539806 (+ 3468us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.539969 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:13.553056 1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1544) took 1244 ms (client timeout 1583 ms). Trace:
I20260927 23:31:13.553267 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1524) took 2157 ms. Trace:
W20260927 23:31:13.553552 1728 rpcz_store.cc:269] 0927 23:31:12.309098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.455558 (+146460us) service_pool.cc:227] Handling call
0927 23:31:13.552863 (+1097305us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54003c4e54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.553003 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300133632}
I20260927 23:31:13.535569 1657 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:13.555088 1728 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260927 23:31:13.555649 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1527) took 2127 ms. Trace:
I20260927 23:31:13.555963 1728 rpcz_store.cc:276] 0927 23:31:11.430110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.554556 (+2124446us) service_pool.cc:227] Handling call
0927 23:31:13.554886 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:13.555438 (+ 552us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.555607 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.553773 1732 rpcz_store.cc:276] 0927 23:31:11.396066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.537925 (+2141859us) service_pool.cc:227] Handling call
0927 23:31:13.538244 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.552980 (+ 14736us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.553210 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.557185 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.556968 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.558153 1732 raft_consensus.cc:2396] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:13.565534 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.566151 1732 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:13.568509 3473 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:13.568792 3502 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.569985 1733 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.572990 1731 raft_consensus.cc:1226] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:13.573293 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.573676 1728 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:13.574476 3502 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.575146 1735 raft_consensus.cc:2396] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:13.573941 1733 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:13.574949 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.575232 2258 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:13.577035 1728 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:13.577529 3590 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:13.577952 3590 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.578506 3590 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.579392 1627 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:13.575978 1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1243) took 2056 ms (client timeout 2627 ms). Trace:
I20260927 23:31:13.580013 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.580128 3464 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:13.581570 3464 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.581926 3464 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:13.580338 1735 rpcz_store.cc:269] 0927 23:31:11.519425 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.423058 (+1903633us) service_pool.cc:227] Handling call
0927 23:31:13.575926 (+152868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.585619 3590 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.586366 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:13.588380 1729 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:13.577888 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:13.583300 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:13.588974 1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 232) took 2176 ms (client timeout 2710 ms). Trace:
W20260927 23:31:13.589321 1729 rpcz_store.cc:269] 0927 23:31:11.412515 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.579777 (+2167262us) service_pool.cc:227] Handling call
0927 23:31:13.588929 (+ 9152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.589462 1728 raft_consensus.cc:2396] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:13.589290 1735 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:13.593217 1700 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:13.587913 3524 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:13.595366 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:13.595955 1735 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:13.596551 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1529) took 2144 ms. Trace:
I20260927 23:31:13.595728 3393 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:13.596835 1735 rpcz_store.cc:276] 0927 23:31:11.452136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.591369 (+2139233us) service_pool.cc:227] Handling call
0927 23:31:13.595667 (+ 4298us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.596285 (+ 618us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.596505 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.598848 1728 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:13.600791 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.601394 1733 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:13.598100 3470 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:13.603401 3470 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.603999 3431 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:13.602469 1733 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:13.605674 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 233) took 2112 ms. Trace:
I20260927 23:31:13.598765 1627 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:13.606370 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1229) took 2092 ms. Trace:
I20260927 23:31:13.607023 1733 rpcz_store.cc:276] 0927 23:31:11.514447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.601974 (+2087527us) service_pool.cc:227] Handling call
0927 23:31:13.602248 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.606070 (+ 3822us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.606297 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.606108 1728 rpcz_store.cc:276] 0927 23:31:11.488174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.598270 (+2110096us) service_pool.cc:227] Handling call
0927 23:31:13.598643 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.600193 (+ 1550us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.600412 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.609818 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1241) took 2091 ms. Trace:
I20260927 23:31:13.607450 1627 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:13.611636 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1526) took 2195 ms. Trace:
I20260927 23:31:13.610379 1700 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:13.612862 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:13.613850 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:13.613533 1729 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:13.615464 3492 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:13.617945 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.618264 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.618185 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.618682 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.620182 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.611021 1728 rpcz_store.cc:276] 0927 23:31:11.518827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.609078 (+2090251us) service_pool.cc:227] Handling call
0927 23:31:13.609356 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.609535 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:13.609570 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:13.609606 (+ 36us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:13.609634 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.609686 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.609776 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.624754 2291 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:13.626830 3500 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:13.627157 1729 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260927 23:31:13.627374 3500 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.627831 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1244) took 2099 ms. Trace:
I20260927 23:31:13.628100 1629 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:13.628211 1729 rpcz_store.cc:276] 0927 23:31:11.528313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.624041 (+2095728us) service_pool.cc:227] Handling call
0927 23:31:13.626935 (+ 2894us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:13.627592 (+ 657us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.627782 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.628748 3416 raft_consensus.cc:3032] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:13.627856 1733 raft_consensus.cc:1261] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:13.629925 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1237) took 2112 ms. Trace:
I20260927 23:31:13.630509 1733 rpcz_store.cc:276] 0927 23:31:11.517136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.608118 (+2090982us) service_pool.cc:227] Handling call
0927 23:31:13.627375 (+ 19257us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.629701 (+ 2326us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.629872 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.612913 1736 rpcz_store.cc:276] 0927 23:31:11.416293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.552174 (+2135881us) service_pool.cc:227] Handling call
0927 23:31:13.552525 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.552728 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:13.555076 (+ 2348us) log.cc:842] Serialized 14 byte log entry
0927 23:31:13.610443 (+ 55367us) log.cc:842] Serialized 14 byte log entry
0927 23:31:13.611342 (+ 899us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:13.611394 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:13.611435 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.611489 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.611588 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2164,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":496,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":148,"spinlock_wait_cycles":47488,"thread_start_us":54011,"threads_started":1,"wal-append.queue_time_us":54344}
I20260927 23:31:13.634888 1728 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:13.635607 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1248) took 2106 ms. Trace:
I20260927 23:31:13.636162 1728 rpcz_store.cc:276] 0927 23:31:11.529508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.625070 (+2095562us) service_pool.cc:227] Handling call
0927 23:31:13.626149 (+ 1079us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.635356 (+ 9207us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.635551 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.629108 3416 raft_consensus.cc:729] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:13.636322 3591 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:13.640733 1729 raft_consensus.cc:1261] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:13.637696 3500 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.636389 3525 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260927 23:31:13.639909 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.612507 3524 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.641427 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1251) took 2111 ms. Trace:
I20260927 23:31:13.643412 1729 rpcz_store.cc:276] 0927 23:31:11.530303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.629087 (+2098784us) service_pool.cc:227] Handling call
0927 23:31:13.639392 (+ 10305us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.641160 (+ 1768us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.641372 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.629379 3472 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:13.644502 3472 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:13.645601 3472 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.647289 1733 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260927 23:31:13.647744 1733 pending_rounds.cc:76] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 5
I20260927 23:31:13.648000 1733 pending_rounds.cc:97] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260927 23:31:13.648283 1733 raft_consensus.cc:2866] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:13.625996 3502 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:13.650615 3638 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:13.651505 1736 raft_consensus.cc:1226] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:13.649071 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1253) took 2118 ms. Trace:
W20260927 23:31:13.661561 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 47767.7us
I20260927 23:31:13.663132 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1523) took 2277 ms. Trace:
I20260927 23:31:13.656335 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1533) took 2088 ms. Trace:
I20260927 23:31:13.640939 3591 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:13.550203 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.997308 1736 rpcz_store.cc:276] 0927 23:31:11.568125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.632084 (+2063959us) service_pool.cc:227] Handling call
0927 23:31:13.651329 (+ 19245us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.655973 (+ 4644us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.656270 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.997525 3591 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:13.998008 1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1447) took 1596 ms (client timeout 1856 ms). Trace:
W20260927 23:31:13.998466 1660 rpcz_store.cc:269] 0927 23:31:12.401702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.422254 (+ 20552us) service_pool.cc:227] Handling call
0927 23:31:13.550037 (+1127783us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:13.997949 (+447912us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7822592}
I20260927 23:31:13.999583 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.000175 1660 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.690404 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:13.655825 3638 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.002626 1584 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:14.003947 3638 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.003512 3591 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.654743 3416 consensus_queue.cc:260] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:13.656725 3431 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:13.630369 3464 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.020860 3464 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.698524 3590 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.635036 3492 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.023963 3492 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:13.665971 3502 raft_consensus.cc:686] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:13.636592 2287 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:13.706615 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 67532.8us
I20260927 23:31:13.721257 1732 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:13.764671 3380 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:14.028841 1729 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:13.498740 1658 rpcz_store.cc:276] 0927 23:31:12.401942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.414287 (+ 12345us) service_pool.cc:227] Handling call
0927 23:31:13.448055 (+1033768us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:13.498300 (+ 50245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23500672}
I20260927 23:31:14.028965 3380 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.029786 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:13.790417 1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48984 (request call id 242) took 19853 ms (client timeout 15000 ms). Trace:
I20260927 23:31:13.800209 3613 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.031029 3380 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.843834 3506 consensus_queue.cc:1046] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:14.031260 3613 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.031826 3506 raft_consensus.cc:3032] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:14.032177 3506 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:14.032225 1629 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.033227 3613 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.032996 3506 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:13.844182 1583 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:13.847777 3643 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:13.846948 1551 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.035006 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:14.035187 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 188305us
I20260927 23:31:14.035269 3643 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:14.035666 3643 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:14.035810 1551 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.036010 3643 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.036232 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:14.036543 3510 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:14.036832 2288 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.036963 3510 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:14.037403 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.037405 3510 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.876693 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.878705 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.890923 3393 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46514 (request call id 851) took 14947 ms. Trace:
I20260927 23:31:13.892943 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.039445 3535 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:14.039738 1659 raft_consensus.cc:2362] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4. Re-sending same reply.
I20260927 23:31:14.040012 3535 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:14.040598 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.041293 1659 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:14.040907 3535 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:13.877295 3633 log.cc:924] Time spent T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.265s user 0.004s sys 0.000s
I20260927 23:31:14.041980 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:13.898265 3627 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.286s user 0.002s sys 0.000s
I20260927 23:31:13.908507 1733 rpcz_store.cc:276] 0927 23:31:11.530827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.631470 (+2100643us) service_pool.cc:227] Handling call
0927 23:31:13.643333 (+ 11863us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.648754 (+ 5421us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.649010 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.909072 1627 leader_election.cc:398] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:13.927263 2286 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:13.928606 3472 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:13.939237 3629 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.329s user 0.003s sys 0.000s
I20260927 23:31:13.655339 3500 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:13.949913 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 223) took 2612 ms. Trace:
I20260927 23:31:13.953948 1734 rpcz_store.cc:276] 0927 23:31:11.385547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.616518 (+2230971us) service_pool.cc:227] Handling call
0927 23:31:13.616875 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:13.617080 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:13.617108 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:13.617500 (+ 392us) log.cc:842] Serialized 26 byte log entry
0927 23:31:13.662673 (+ 45173us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:13.662730 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.662822 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:13.662884 (+ 62us) raft_consensus.cc:1671] finished
0927 23:31:13.662901 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.663057 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":44777,"threads_started":1,"wal-append.queue_time_us":26802}
I20260927 23:31:13.955590 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.963330 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:13.969225 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.042747 1659 raft_consensus.cc:2362] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4. Re-sending same reply.
I20260927 23:31:14.049829 1733 raft_consensus.cc:1226] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:14.050057 1627 leader_election.cc:401] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 23:31:14.050511 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 235) took 2073 ms. Trace:
I20260927 23:31:14.050860 1733 rpcz_store.cc:276] 0927 23:31:11.977108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.049373 (+2072265us) service_pool.cc:227] Handling call
0927 23:31:14.049669 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.050268 (+ 599us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.050449 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:14.051481 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.051826 3638 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:14.052121 1655 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:14.052627 3639 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:14.052345 3472 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.053059 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.054612 3472 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.054812 1662 raft_consensus.cc:2362] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6. Re-sending same reply.
I20260927 23:31:14.055593 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.055891 3500 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.056272 3500 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:14.050055 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.056733 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.057809 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.972949 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1530) took 2477 ms. Trace:
I20260927 23:31:14.052240 3638 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:14.058691 1735 rpcz_store.cc:276] 0927 23:31:11.495866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.598725 (+2102859us) service_pool.cc:227] Handling call
0927 23:31:13.639649 (+ 40924us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:13.639861 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:13.639948 (+ 87us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:13.642522 (+ 2574us) log.cc:842] Serialized 26 byte log entry
0927 23:31:13.972727 (+330205us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:13.972752 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.972798 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:13.972828 (+ 30us) raft_consensus.cc:1671] finished
0927 23:31:13.972834 (+ 6us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.972910 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":322427,"threads_started":1,"wal-append.queue_time_us":53512}
I20260927 23:31:14.059589 3638 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:13.980413 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 234) took 2061 ms. Trace:
I20260927 23:31:14.060813 1728 rpcz_store.cc:276] 0927 23:31:11.579740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.639918 (+2060178us) service_pool.cc:227] Handling call
0927 23:31:13.640184 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.640369 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:13.640403 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:13.640453 (+ 50us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:13.640493 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.640553 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:13.640658 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:13.981642 1484 catalog_manager.cc:5495] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.062551 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.062820 1735 raft_consensus.cc:1226] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:14.063582 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.063150 1484 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.064272 1663 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:14.064980 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:13.983076 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:13.983722 1588 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:13.977532 3468 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:13.984666 1579 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:14.066614 1728 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 23:31:13.989352 1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1532) took 2422 ms (client timeout 1927 ms). Trace:
W20260927 23:31:14.067251 1730 rpcz_store.cc:269] 0927 23:31:11.566746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.192568 (+625822us) service_pool.cc:227] Handling call
0927 23:31:13.989154 (+1796586us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54003c3054. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.989285 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1151796992}
I20260927 23:31:13.990621 1737 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:14.068455 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1528) took 2620 ms. Trace:
I20260927 23:31:13.985842 3470 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:13.998486 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:14.006229 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.005592 3591 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:13.814220 1554 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.017765 3431 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:14.075812 3639 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3.
I20260927 23:31:14.064698 1484 catalog_manager.cc:5495] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 4, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.039206 3393 rpcz_store.cc:276] 0927 23:30:58.943469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:58.946308 (+ 2839us) service_pool.cc:227] Handling call
0927 23:30:59.265192 (+318884us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b58005405b0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f97509eaf63 00007f9750a4eaba 00007f9750a6aa7a 00007f9750a6bada 00007f9750a6ac8b 00007f9750a76084 00007f9750a75743 00007f9750a6dc1a 00007f97559b16d9 00007f974f09b184
0927 23:30:59.317035 (+ 51843us) log.cc:842] Serialized 480 byte log entry
0927 23:30:59.958496 (+641461us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b18003561a0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:00.207774 (+249278us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b58006f1ab0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4af0 00007f97509e9834 00007f97509ee771 00007f97509ee6f9 00007f97509ee689 00007f97509ee651 00007f97509ed81d 00007f9756dab204 00007f9756dab039 00007f974db476b6 00007f974db4ab11
0927 23:31:13.890795 (+13683021us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9,"raft.queue_time_us":3637,"raft.run_cpu_time_us":1796,"raft.run_wall_time_us":297643,"spinlock_wait_cycles":404494208,"thread_start_us":38981,"threads_started":1,"wal-append.queue_time_us":109410,"wal-append.run_cpu_time_us":1848,"wal-append.run_wall_time_us":1084478}
I20260927 23:31:14.072284 3590 consensus_queue.cc:1046] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260927 23:31:14.081120 3590 raft_consensus.cc:3032] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:14.081467 3590 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:14.082535 1736 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.083168 1730 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260927 23:31:14.082535 3644 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:14.072382 1658 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:14.081135 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 266972us
I20260927 23:31:14.085744 3644 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:14.081674 1484 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 3, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:31:14.030457 1486 rpcz_store.cc:269] 0927 23:30:53.937012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:30:53.937194 (+ 182us) service_pool.cc:227] Handling call
0927 23:31:13.790203 (+19853009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:05.241535 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:05.368721 (+127186us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:05.378735 (+ 10014us) write_op.cc:435] Acquired schema lock
0927 23:31:05.385668 (+ 6933us) tablet.cc:660] Decoding operations
0927 23:31:05.475443 (+ 89775us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:05.475556 (+ 113us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:05.475597 (+ 41us) tablet.cc:683] Acquiring locks for 3 operations
0927 23:31:05.476078 (+ 481us) tablet.cc:699] Row locks acquired
0927 23:31:05.476106 (+ 28us) write_op.cc:260] PREPARE: finished
0927 23:31:05.476314 (+ 208us) write_op.cc:270] Start()
0927 23:31:05.476444 (+ 130us) write_op.cc:276] Timestamp: P: 1790551865476264 usec, L: 0
0927 23:31:05.476464 (+ 20us) op_driver.cc:331] REPLICATION: starting
0927 23:31:05.477262 (+ 798us) log.cc:842] Serialized 1290 byte log entry
0927 23:31:07.004367 (+1527105us) op_driver.cc:444] REPLICATION: finished
0927 23:31:07.724623 (+720256us) write_op.cc:301] APPLY: starting
0927 23:31:07.811094 (+ 86471us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:07.854468 (+ 43374us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:07.943763 (+ 89295us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:08.497536 (+553773us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:08.548984 (+ 51448us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 23:31:08.549052 (+ 68us) write_op.cc:312] APPLY: finished
0927 23:31:08.818087 (+269035us) log.cc:842] Serialized 35 byte log entry
0927 23:31:10.952295 (+2134208us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f974db295fe 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb
0927 23:31:11.136976 (+184681us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:11.158060 (+ 21084us) write_op.cc:454] Released schema lock
0927 23:31:11.158134 (+ 74us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:11.158627 (+ 493us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:11.158658 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2618497,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":708543,"apply.run_cpu_time_us":37224,"apply.run_wall_time_us":3445682,"num_ops":3,"prepare.queue_time_us":100130,"prepare.run_cpu_time_us":24949,"prepare.run_wall_time_us":409306,"raft.queue_time_us":859600,"raft.run_cpu_time_us":66960,"raft.run_wall_time_us":634944,"replication_time_us":1527651,"spinlock_wait_cycles":8253312,"thread_start_us":3761802,"threads_started":5,"wal-append.queue_time_us":2711764,"wal-append.run_cpu_time_us":40495,"wal-append.run_wall_time_us":2933173}]]}
I20260927 23:31:14.087742 1484 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.088510 3431 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.054384 1731 rpcz_store.cc:276] 0927 23:31:11.337089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.518244 (+2181155us) service_pool.cc:227] Handling call
0927 23:31:13.572825 (+ 54581us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:13.573355 (+ 530us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:13.949624 (+376269us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b54001dea50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:13.949831 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124583168}
I20260927 23:31:14.049829 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.089742 3431 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.089977 1656 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.025866 3502 consensus_queue.cc:237] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.027695 3117 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.029413 1732 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:13.813256 3616 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:13.862205 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 1.7678e+06us
I20260927 23:31:14.091620 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1536) took 2211 ms. Trace:
I20260927 23:31:14.091462 3616 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.050462 1734 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:14.049539 3643 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.021627 3416 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:14.066295 1581 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.060098 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1534) took 2466 ms. Trace:
I20260927 23:31:14.069811 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:14.093529 3468 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.102478 3468 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:14.068766 1737 rpcz_store.cc:276] 0927 23:31:11.448011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.990116 (+2542105us) service_pool.cc:227] Handling call
0927 23:31:13.990448 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.068250 (+ 77802us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.068409 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:14.070055 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:13.907088 3524 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:14.107353 1484 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:14.108711 1484 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:14.034137 3506 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:14.112780 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.113528 1737 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:14.113183 3638 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.116398 1659 raft_consensus.cc:2354] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:14.116432 3638 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.038645 1654 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.096769 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1555) took 1059 ms. Trace:
I20260927 23:31:14.120566 1734 rpcz_store.cc:276] 0927 23:31:13.037449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.050050 (+1012601us) service_pool.cc:227] Handling call
0927 23:31:14.050290 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.096563 (+ 46273us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.096721 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:14.071390 2286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 446) took 1718 ms (client timeout 1777 ms). Trace:
I20260927 23:31:14.072834 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:14.098986 1729 rpcz_store.cc:276] 0927 23:31:11.593697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:13.649879 (+2056182us) service_pool.cc:227] Handling call
0927 23:31:14.028676 (+378797us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.058408 (+ 29732us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.059937 (+ 1529us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54001d1d50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:14.060053 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":374837120}
I20260927 23:31:14.123574 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.124339 3503 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:14.121495 2286 rpcz_store.cc:269] 0927 23:31:12.352925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.355051 (+ 2126us) service_pool.cc:227] Handling call
0927 23:31:13.499473 (+1144422us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:14.071211 (+571738us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54006419d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:14.071320 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310501120}
I20260927 23:31:14.125363 1551 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.097684 3616 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.118713 1579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:14.126853 1579 raft_consensus.cc:595] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.127195 1579 raft_consensus.cc:603] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:14.095155 2901 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.128127 1700 leader_election.cc:398] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:14.127972 2901 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.091872 3117 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.129268 1733 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 23:31:14.128436 1700 leader_election.cc:401] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:14.130916 3613 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.133266 1579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
W20260927 23:31:14.072717 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 163790us
I20260927 23:31:14.133914 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.136894 3590 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.054117 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.010995 1660 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:14.140117 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.092701 1732 rpcz_store.cc:276] 0927 23:31:11.880822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.028937 (+2148115us) service_pool.cc:227] Handling call
0927 23:31:14.029232 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.091347 (+ 62115us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.091553 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:14.141474 1655 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:14.143592 3117 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:14.144380 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.152215 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.152345 3464 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:14.152890 3464 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:14.153324 3464 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.153350 3492 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.153820 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.154011 3492 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.154623 1730 raft_consensus.cc:2396] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:14.155974 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:14.155885 3492 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.157066 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.157639 1730 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:14.146052 1662 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 23:31:14.158490 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:14.071295 3510 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.159507 1736 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:14.151628 1660 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:14.160280 1661 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:14.153401 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.175313 3503 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:14.175868 3503 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:14.176415 1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1457) took 1522 ms (client timeout 2013 ms). Trace:
I20260927 23:31:14.176613 3503 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:14.178747 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 53449.4us
I20260927 23:31:14.163357 1654 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:14.179363 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.150943 2901 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.180536 3616 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:14.181030 3616 raft_consensus.cc:686] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:14.181875 3616 consensus_queue.cc:237] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.182452 2257 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:14.183722 1552 connection.cc:232] RPC call timeout handler was delayed by -1.9754199609999432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2589s for a response
I20260927 23:31:14.184973 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.186105 1551 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.101964 2288 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:14.186424 3473 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:14.186865 3393 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:14.187386 3393 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:14.188177 3393 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:14.189792 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 74877.9us
I20260927 23:31:14.135409 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.191725 3134 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:14.192368 1663 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:14.193396 1663 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:14.194902 3473 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:14.195818 1732 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:14.198273 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:14.199054 1732 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:14.199891 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.201128 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.201702 1734 raft_consensus.cc:2362] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6. Re-sending same reply.
I20260927 23:31:14.202657 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.203706 3590 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:14.100243 3643 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.168794 1656 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
W20260927 23:31:14.170547 1702 leader_election.cc:334] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.906s (SENT)
I20260927 23:31:14.109877 1484 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 5, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 5 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:14.209028 1656 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:14.209578 1484 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.211309 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.118417 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:14.128631 1628 leader_election.cc:334] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.742s (SENT)
I20260927 23:31:14.171285 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 239) took 1092 ms. Trace:
W20260927 23:31:14.213356 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 84751.5us
I20260927 23:31:14.213770 1733 rpcz_store.cc:276] 0927 23:31:13.078858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.051581 (+972723us) service_pool.cc:227] Handling call
0927 23:31:14.051824 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.129677 (+ 77853us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.171094 (+ 41417us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b540011d354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:14.171238 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177361280}
I20260927 23:31:14.166589 1657 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:14.167820 1658 raft_consensus.cc:2396] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260927 23:31:14.184860 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.190447 3525 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:14.168126 1581 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:14.175338 1731 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:14.218395 1579 raft_consensus.cc:595] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.212327 1484 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 3, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:14.218771 1579 raft_consensus.cc:603] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:14.219727 1658 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260927 23:31:14.176805 1661 rpcz_store.cc:269] 0927 23:31:12.654108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:12.671469 (+ 17361us) service_pool.cc:227] Handling call
0927 23:31:14.006082 (+1334613us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:14.176269 (+170187us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b54000dc0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:14.176367 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30820992}
I20260927 23:31:14.220063 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:14.210664 1552 connection.cc:232] RPC call timeout handler was delayed by -1.9467481190000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260927 23:31:14.064774 1735 raft_consensus.cc:1261] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:14.169070 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:14.222033 2259 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:14.212356 1736 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:14.223306 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:14.241835 1662 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:14.243091 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 74118.6us
I20260927 23:31:14.243868 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.243481 1484 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:14.249899 1552 connection.cc:232] RPC call timeout handler was delayed by -1.8627045949999683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2263s for a response
I20260927 23:31:14.252107 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:14.252841 1737 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.254199 1700 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:14.256317 1628 leader_election.cc:334] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.800s (SENT)
I20260927 23:31:14.256852 1628 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:14.258293 1628 leader_election.cc:334] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.827s (SENT)
I20260927 23:31:14.258762 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.273422 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.274592 1700 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.275956 3472 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:14.320670 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 99619.4us
I20260927 23:31:14.320695 1658 raft_consensus.cc:1261] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 23:31:14.320395 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 100431us
I20260927 23:31:14.180559 1654 raft_consensus.cc:1261] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260927 23:31:14.329100 1654 pending_rounds.cc:76] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 2
I20260927 23:31:14.329378 1654 pending_rounds.cc:97] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260927 23:31:14.329665 1654 raft_consensus.cc:2866] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:14.331241 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.325294 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:14.218433 3506 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:14.324672 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 102726us
I20260927 23:31:14.272853 3524 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:14.250681 3638 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:14.272691 3525 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:14.277063 3591 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:14.347939 3500 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.326421 3642 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:14.351749 1702 leader_election.cc:334] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.087s (SENT)
I20260927 23:31:14.320362 3472 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:14.326156 3470 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:14.357882 1702 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.324261 3380 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:14.336274 3416 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:14.239540 3468 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.346158 3654 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:14.349718 1655 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260927 23:31:14.344666 3431 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:14.350723 3524 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:14.395361 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 137100us
I20260927 23:31:14.397503 1700 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.343833 3516 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260927 23:31:14.397084 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.400458 1656 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:14.401105 1700 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.362221 3638 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:14.362963 3472 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:14.410777 1628 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.865s (SENT)
I20260927 23:31:14.362443 3525 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:14.347311 3506 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.374465 3117 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:14.412873 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.349048 3473 consensus_queue.cc:1046] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260927 23:31:14.399729 3642 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.411789 1579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:14.426672 1579 raft_consensus.cc:595] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.427121 1579 raft_consensus.cc:603] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:14.397256 3524 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:14.375280 3380 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:14.425271 1702 leader_election.cc:334] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.144s (SENT)
I20260927 23:31:14.431612 1702 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.377460 3470 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:14.401484 3431 raft_consensus.cc:686] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:14.413283 3439 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:14.377825 3510 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.441838 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:31:14.422968 3651 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:14.414487 3638 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.370913 3134 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
W20260927 23:31:14.464512 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 278447us
I20260927 23:31:14.412220 3492 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.410257 3639 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:14.460626 3439 raft_consensus.cc:686] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:14.455559 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:14.466281 1551 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.438844 3117 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:14.439044 3613 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:14.436620 3380 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:14.413450 3502 consensus_queue.cc:1046] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260927 23:31:14.465202 3638 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:14.457976 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 126850us
I20260927 23:31:14.482007 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:14.487007 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 148155us
I20260927 23:31:14.492244 1629 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:14.495847 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 94844.3us
I20260927 23:31:14.468598 1654 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:31:14.449401 3642 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.471619 2901 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:14.547875 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.550958 1661 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.564109 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:14.469936 3492 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.469547 3663 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260927 23:31:14.567111 3591 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:14.605751 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 112274us
I20260927 23:31:14.602559 3439 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.689405 3525 raft_consensus.cc:3032] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:14.689863 3525 raft_consensus.cc:729] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:14.688292 3660 log.cc:924] Time spent T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260927 23:31:14.696688 1737 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:14.495610 3592 consensus_queue.cc:1035] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
W20260927 23:31:14.705770 1517 debug-util.cc:398] Leaking SignalData structure 0x7b080003efa0 after lost signal to thread 399
I20260927 23:31:14.591399 3516 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260927 23:31:14.467160 3664 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:14.703655 1661 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:14.680932 3510 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.680294 3468 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.720285 3510 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.723337 3510 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.725313 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.726598 1551 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:14.728739 1517 debug-util.cc:398] Leaking SignalData structure 0x7b08000346a0 after lost signal to thread 1520
I20260927 23:31:14.694242 3613 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.572968 3470 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.571861 3431 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.692610 3591 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:14.707343 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:14.706898 3506 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:14.694612 3473 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260927 23:31:14.491488 3464 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:14.777648 1552 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.004s sys 0.022s
W20260927 23:31:14.779470 1552 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.004s sys 0.022s
W20260927 23:31:14.779908 1552 connection.cc:232] RPC call timeout handler was delayed by -1.5740930370000115s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
W20260927 23:31:14.780288 1552 connection.cc:232] RPC call timeout handler was delayed by -1.5680930710000212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260927 23:31:14.780717 1552 connection.cc:232] RPC call timeout handler was delayed by -1.3161717569999496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
I20260927 23:31:14.781031 3464 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:14.781064 1552 connection.cc:232] RPC call timeout handler was delayed by -1.2548107279999954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1559s for a response
W20260927 23:31:14.781818 1552 connection.cc:232] RPC call timeout handler was delayed by -1.1942164160000175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17390000000000003s for a response
I20260927 23:31:14.782572 1660 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:14.784010 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 197) took 1162 ms. Trace:
I20260927 23:31:14.784423 1660 rpcz_store.cc:276] 0927 23:31:13.621679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.174513 (+552834us) service_pool.cc:227] Handling call
0927 23:31:14.174798 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.783083 (+608285us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.783740 (+ 657us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b5400352354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:14.783942 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1397209472}
I20260927 23:31:14.789361 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:14.785980 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:14.792622 1660 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:14.732682 3500 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:14.808044 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 117933us
I20260927 23:31:14.767748 3510 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:14.775682 3502 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.814378 1660 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
W20260927 23:31:14.816331 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 491105us
I20260927 23:31:14.822688 1551 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:14.470243 3639 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.824467 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:14.799710 2286 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:14.839285 2260 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.018s sys 0.073s
I20260927 23:31:14.832077 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.842389 1627 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:14.846928 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:14.847707 2293 raft_consensus.cc:2379] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:14.848256 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:14.848845 2291 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:14.851445 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:14.770682 3613 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.853341 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.855090 3439 consensus_queue.cc:1046] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260927 23:31:14.841467 1517 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcea0 after lost signal to thread 1612
I20260927 23:31:14.743033 3525 consensus_queue.cc:260] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:14.830217 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 185) took 1211 ms. Trace:
I20260927 23:31:14.817580 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:14.902442 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 53206.9us
I20260927 23:31:14.839198 3510 raft_consensus.cc:686] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:14.917309 3590 consensus_meta.cc:287] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.712s user 0.016s sys 0.001s
W20260927 23:31:14.967350 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 114054us
I20260927 23:31:14.905700 3503 raft_consensus.cc:2741] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260927 23:31:14.974941 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:14.905616 3393 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:14.986588 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.831868 3502 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:14.928794 3473 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:14.898702 3506 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:14.843053 3464 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.130s
I20260927 23:31:14.847982 3468 consensus_queue.cc:1046] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.135s
I20260927 23:31:14.851119 3664 consensus_queue.cc:1035] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.144s
I20260927 23:31:14.966218 3535 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:14.995622 3416 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.002s sys 0.008s
I20260927 23:31:14.973762 3492 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:15.002977 1517 debug-util.cc:398] Leaking SignalData structure 0x7b080038e480 after lost signal to thread 1687
I20260927 23:31:15.014686 3492 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:15.015076 3673 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260927 23:31:15.015226 3492 raft_consensus.cc:686] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:15.015564 1517 debug-util.cc:398] Leaking SignalData structure 0x7b080025b3c0 after lost signal to thread 1761
I20260927 23:31:15.016253 3492 consensus_queue.cc:237] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:15.017016 1517 debug-util.cc:398] Leaking SignalData structure 0x7b0800234b80 after lost signal to thread 2318
W20260927 23:31:15.017349 1517 debug-util.cc:398] Leaking SignalData structure 0x7b08003aad80 after lost signal to thread 2319
I20260927 23:31:15.017851 3672 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:31:15.018355 3672 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:15.018748 3672 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:31:15.018996 3613 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:15.019515 3613 raft_consensus.cc:3032] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:15.019845 3613 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:14.981467 1654 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:14.984547 3393 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:15.009297 1660 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:15.002878 3642 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:14.990060 3687 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:15.056728 3674 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.342s user 0.000s sys 0.001s
I20260927 23:31:14.813673 3676 consensus_queue.cc:1035] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.256s
W20260927 23:31:15.013335 3416 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.594s user 0.002s sys 0.009s
I20260927 23:31:14.921048 3591 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.021170 3506 raft_consensus.cc:686] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:15.074298 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:14.922883 1655 rpcz_store.cc:276] 0927 23:31:13.614811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.143590 (+528779us) service_pool.cc:227] Handling call
0927 23:31:14.273603 (+130013us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:14.396121 (+122518us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:14.805564 (+409443us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001cfcd0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:14.826027 (+ 20463us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265819520}
W20260927 23:31:15.079416 1702 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.775s (SENT) [suppressed 70 similar messages]
I20260927 23:31:15.061118 3639 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:15.002907 3473 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.083352 1660 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:15.085992 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 379741us
I20260927 23:31:15.061843 3393 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.097370 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:15.100502 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 177163us
I20260927 23:31:15.085537 3535 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:15.073465 3642 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260927 23:31:15.141942 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:15.082334 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:15.116482 3639 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:15.069929 3510 consensus_queue.cc:237] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.143695 3535 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:15.148211 3664 raft_consensus.cc:3032] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:15.088378 3680 consensus_queue.cc:1046] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20260927 23:31:15.165373 3703 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:15.180732 3703 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:15.190455 3703 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:15.191926 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:15.081400 3613 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.160094 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:15.196595 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 210188us
W20260927 23:31:15.170198 3651 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.016s sys 0.015s
I20260927 23:31:15.171140 3535 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:15.169206 3464 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:15.193041 3671 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260927 23:31:15.174399 3664 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:15.178295 1655 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 23:31:15.199357 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.118992 3685 consensus_queue.cc:1035] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20260927 23:31:15.227634 3685 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:15.228247 3685 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.199640 3472 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.768s user 0.009s sys 0.002s
W20260927 23:31:15.238077 3679 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
W20260927 23:31:15.198005 3651 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.017s sys 0.015s
I20260927 23:31:15.203435 3502 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:15.169041 3639 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.161332 3506 consensus_queue.cc:237] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.207307 3669 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:15.242358 3613 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.247658 1554 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.218778 3464 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.241786 3651 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:15.273561 3643 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:15.199471 3709 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
W20260927 23:31:15.246892 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 47766.4us
W20260927 23:31:15.249245 3380 consensus_meta.cc:287] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.754s user 0.010s sys 0.001s
I20260927 23:31:15.239488 3472 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.270233 3700 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260927 23:31:15.272358 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 161364us
I20260927 23:31:15.206303 3616 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260927 23:31:15.245677 3707 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:15.277417 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.250098 3416 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:15.254948 1654 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 23:31:15.258265 3698 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:15.284003 3524 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.830s user 0.011s sys 0.000s
I20260927 23:31:15.292786 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.295444 3707 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.288625 1658 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:15.300995 1702 leader_election.cc:334] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.029s (SENT)
I20260927 23:31:15.307857 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.276175 3473 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.301843 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.310515 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.303555 1732 raft_consensus.cc:2379] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 3: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:15.311123 1729 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:15.311866 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.312624 1732 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.238091 3664 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.304054 1731 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:15.320390 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.321985 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.322067 1732 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:15.325285 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.326917 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.287282 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:15.338431 1728 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.282299 3643 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
W20260927 23:31:15.354246 2294 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.002s sys 0.015s
I20260927 23:31:15.356379 1734 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:15.358464 2294 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.518s user 0.002s sys 0.016s
I20260927 23:31:15.362908 1728 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:15.282183 3380 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.367259 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 79552.4us
I20260927 23:31:15.256307 3669 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.319317 3464 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.297948 3416 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.307914 3524 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.367201 3664 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:15.364961 3431 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:15.393208 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.386394 1735 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260927 23:31:15.364751 1655 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:15.367342 3685 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:15.397308 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 112766us
I20260927 23:31:15.372833 3709 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:15.381165 3502 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:15.384320 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:15.415382 1663 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:15.415719 1552 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.291848 3651 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.386237 3717 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:15.391804 3693 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:15.400771 3464 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:15.421801 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.404644 3685 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:15.442692 3639 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:15.443888 3707 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:15.415556 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:15.455029 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 97125.5us
W20260927 23:31:15.457335 1702 leader_election.cc:334] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.557s (SENT)
I20260927 23:31:15.463419 1702 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.463745 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.466668 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.467363 1737 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:15.468674 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.471715 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.403116 3431 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.472947 1737 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:15.487471 1627 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.488376 1627 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.431013 3709 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260927 23:31:15.456185 3506 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:15.455754 2901 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:15.494416 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:15.492882 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.449024 3643 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260927 23:31:15.469892 3590 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:15.501964 1733 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:15.498296 3639 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:15.523032 3639 raft_consensus.cc:686] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:15.524063 2287 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:15.490687 3472 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:15.525877 3472 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:15.526396 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:15.527732 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:15.528482 1654 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:15.488487 3638 consensus_meta.cc:287] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.926s user 0.012s sys 0.000s
I20260927 23:31:15.529886 1588 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:15.485013 3707 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:15.523077 3669 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:15.531440 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.430711 3717 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.512131 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.484874 3464 raft_consensus.cc:686] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:15.489020 3714 log.cc:924] Time spent T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
I20260927 23:31:15.490309 3642 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:15.540563 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:15.542430 1628 leader_election.cc:334] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.586s (SENT)
I20260927 23:31:15.605175 1583 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
W20260927 23:31:15.605194 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 93161.3us
W20260927 23:31:15.543684 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 128033us
I20260927 23:31:15.605195 1628 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:15.607017 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64637.1us
I20260927 23:31:15.607784 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.488794 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:15.609422 2288 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:15.461669 3685 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.550096 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:15.599189 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 67836.6us
I20260927 23:31:15.532987 3590 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:15.599903 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 69037.9us
I20260927 23:31:15.529533 3705 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:15.516930 2901 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:15.455981 3693 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.539709 3643 consensus_queue.cc:1225] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer a2a3317e8da742d9aefe4c251abd4566 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260927 23:31:15.587308 1702 leader_election.cc:334] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.661s (SENT)
W20260927 23:31:15.689514 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 145259us
I20260927 23:31:15.543938 3117 raft_consensus.cc:3032] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:15.687162 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 144039us
I20260927 23:31:15.698590 1656 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:31:15.493592 3712 consensus_queue.cc:1046] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.118s
I20260927 23:31:15.597873 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:15.755210 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:15.692394 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 84714.8us
I20260927 23:31:15.759511 1552 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:15.759104 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 171855us
I20260927 23:31:15.600116 3613 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:15.610502 3707 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:15.761662 1702 leader_election.cc:334] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.704s (SENT)
I20260927 23:31:15.807580 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:15.809939 1733 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.539837 3638 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.807600 1702 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:15.810990 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 49366.9us
I20260927 23:31:15.695629 3616 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:15.692341 3700 log.cc:924] Time spent T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260927 23:31:15.687042 3380 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:15.814306 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 54892.2us
I20260927 23:31:15.701360 3702 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:15.702248 3524 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.702867 3705 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:15.807294 3729 log.cc:924] Time spent T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.332s user 0.001s sys 0.000s
I20260927 23:31:15.691200 3718 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:15.698902 3590 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:15.696341 3416 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.815761 1703 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.524081 3639 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.754995 3117 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:15.755870 3680 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:15.738883 3725 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.192s
I20260927 23:31:15.826637 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.903209 1737 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:15.854349 3705 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.829439 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 250) took 1337 ms. Trace:
I20260927 23:31:15.905730 1731 rpcz_store.cc:276] 0927 23:31:14.492503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.493298 (+ 795us) service_pool.cc:227] Handling call
0927 23:31:14.604330 (+111032us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:15.330552 (+726222us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:15.829209 (+498657us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b5400148554. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:15.829381 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1412891904}
I20260927 23:31:15.833563 2901 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:15.836030 3688 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.238s user 0.000s sys 0.000s
W20260927 23:31:15.847184 3722 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.397s user 0.002s sys 0.000s
W20260927 23:31:15.847332 3720 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.417s user 0.000s sys 0.001s
I20260927 23:31:15.847565 3687 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 23:31:15.848038 3393 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.729s user 0.004s sys 0.008s
I20260927 23:31:15.912245 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 278) took 1693 ms. Trace:
I20260927 23:31:15.912195 3393 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.912662 1584 rpcz_store.cc:276] 0927 23:31:14.218740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.218860 (+ 120us) service_pool.cc:227] Handling call
0927 23:31:14.219158 (+ 298us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:14.219443 (+ 285us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:14.277772 (+ 58329us) log.cc:842] Serialized 14 byte log entry
0927 23:31:14.799246 (+521474us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:15.910457 (+1111211us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b2c00029428. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:15.910850 (+ 393us) log.cc:842] Serialized 26 byte log entry
0927 23:31:15.911005 (+ 155us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:15.911044 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:15.911114 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:15.911968 (+ 854us) raft_consensus.cc:1671] finished
0927 23:31:15.912058 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:15.912193 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49190,"prepare.queue_time_us":58180,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":27722,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":86462,"spinlock_wait_cycles":12771584,"thread_start_us":456415,"threads_started":1,"wal-append.queue_time_us":575549}
W20260927 23:31:15.848827 3699 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20260927 23:31:15.850163 1659 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:15.850817 1729 consensus_meta.cc:287] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.539s user 0.010s sys 0.000s
I20260927 23:31:15.915596 3717 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.875203 3709 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:15.823167 3535 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.605s user 0.005s sys 0.005s
I20260927 23:31:15.703251 3719 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.148s
W20260927 23:31:15.894753 1628 leader_election.cc:334] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.665s (SENT)
I20260927 23:31:15.924913 1628 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.897787 1733 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:15.875540 2286 consensus_meta.cc:287] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.976s user 0.010s sys 0.003s
I20260927 23:31:15.818740 3730 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:15.827211 3500 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:15.872330 3431 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:15.697324 3703 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.897087 3718 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:15.907866 2901 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:15.990219 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:15.915522 1729 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:15.812201 3669 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:15.837304 1654 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.920401 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 104717us
I20260927 23:31:15.921152 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:15.966424 3717 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:15.964641 1628 leader_election.cc:334] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.736s (SENT)
W20260927 23:31:15.999167 1701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.003s sys 0.008s
W20260927 23:31:15.999464 1701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.003s sys 0.008s
I20260927 23:31:15.969283 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.963477 2286 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:15.974707 3500 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.977136 3731 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260927 23:31:15.989084 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 165257us
I20260927 23:31:15.990705 3393 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:15.993623 1551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.000s sys 0.010s
W20260927 23:31:16.054936 1551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s user 0.000s sys 0.011s
W20260927 23:31:16.055230 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 686894us
I20260927 23:31:15.963325 3380 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:15.816982 3707 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:15.999012 1628 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:15.991091 3718 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:15.998755 1702 leader_election.cc:334] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.856s (SENT)
I20260927 23:31:16.060658 1702 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:16.013664 1627 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:15.851202 3464 consensus_queue.cc:237] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:15.923357 3510 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:16.066187 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 55892.3us
I20260927 23:31:16.015388 3727 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:16.059656 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:16.054484 3672 consensus_meta.cc:287] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.035s user 0.013s sys 0.000s
W20260927 23:31:16.057170 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 92562.7us
I20260927 23:31:16.014743 3739 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:16.061992 1551 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:16.063647 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50070.2us
I20260927 23:31:15.904632 3680 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:15.999871 3134 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.007s sys 0.004s
W20260927 23:31:16.069097 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 70380.4us
I20260927 23:31:16.059181 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 381) took 1355 ms. Trace:
W20260927 23:31:16.079108 1702 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.575s (SENT)
I20260927 23:31:16.079659 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:15.963613 3535 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.057525 3711 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:16.065440 3698 consensus_meta.cc:287] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.755s user 0.006s sys 0.004s
I20260927 23:31:16.067745 3717 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 23:31:16.117650 1702 leader_election.cc:334] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.883s (SENT)
I20260927 23:31:16.122396 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:16.069521 3702 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election won for term 3
W20260927 23:31:16.073493 3734 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
W20260927 23:31:16.080838 3134 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.594s user 0.008s sys 0.004s
I20260927 23:31:16.124522 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:16.127755 1702 leader_election.cc:334] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.603s (SENT)
I20260927 23:31:16.128229 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:16.077775 3638 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:16.131163 3738 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 23:31:16.085373 2286 rpcz_store.cc:276] 0927 23:31:14.702794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.707054 (+ 4260us) service_pool.cc:227] Handling call
0927 23:31:16.053368 (+1346314us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:16.085518 3685 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.085116 3725 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:16.135404 3725 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:16.135824 3725 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:16.136567 3736 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260927 23:31:16.138511 3737 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260927 23:31:16.077430 3727 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.139256 3500 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.011603 3117 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.140242 3117 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:16.140182 3681 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260927 23:31:16.141012 3727 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.142082 3703 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.122740 3745 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:16.117172 3739 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:16.143795 3739 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.074153 3380 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:16.134771 3721 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.238s user 0.000s sys 0.001s
I20260927 23:31:16.147226 1730 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:16.086767 3524 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:16.071208 3510 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.085186 3393 consensus_queue.cc:1046] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20260927 23:31:16.073868 3672 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.088380 2259 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:16.229959 1628 leader_election.cc:334] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.572s (SENT)
I20260927 23:31:16.291601 1628 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:16.087844 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:16.221805 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 139197us
W20260927 23:31:16.294432 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 206648us
W20260927 23:31:16.303694 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 490158us
W20260927 23:31:16.197311 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 130372us
I20260927 23:31:16.304738 1629 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:16.081456 3746 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:16.296685 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 214139us
I20260927 23:31:16.124632 3698 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:16.309234 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 79330us
I20260927 23:31:16.298411 1730 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:16.147606 3728 log.cc:924] Time spent T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
W20260927 23:31:16.155422 3726 log.cc:924] Time spent T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
W20260927 23:31:16.309185 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 220852us
I20260927 23:31:16.313504 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.314376 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:16.314710 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:16.321259 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:16.315026 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:16.322148 1702 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.927s (SENT)
I20260927 23:31:16.317464 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:16.311795 1628 leader_election.cc:334] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.847s (SENT)
I20260927 23:31:16.322930 1628 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:16.323735 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.324014 1588 raft_consensus.cc:2354] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:16.162904 3669 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:16.244757 3745 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.245398 3693 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.155437 3715 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:16.277443 3535 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.214387 3134 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.162966 3702 raft_consensus.cc:686] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:16.222280 3751 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.328765 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.353058 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.364336 1587 raft_consensus.cc:2354] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 7.
I20260927 23:31:16.358116 1586 raft_consensus.cc:1261] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 23:31:16.364861 1586 pending_rounds.cc:76] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 1
I20260927 23:31:16.365175 1586 pending_rounds.cc:97] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 23:31:16.365458 1586 raft_consensus.cc:2866] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:16.307757 3717 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260927 23:31:16.310559 3506 consensus_meta.cc:287] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.792s user 0.007s sys 0.003s
I20260927 23:31:16.325490 3748 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:16.124897 3711 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.383139 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1280) took 1091 ms. Trace:
W20260927 23:31:16.383744 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 71994.3us
I20260927 23:31:16.384826 3654 raft_consensus.cc:3032] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:16.385246 3654 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:16.386029 3654 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.387217 3654 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:16.383621 1728 rpcz_store.cc:276] 0927 23:31:15.292123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.301526 (+ 9403us) service_pool.cc:227] Handling call
0927 23:31:16.382909 (+1081383us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b54001ddb54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:16.383063 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63131392}
I20260927 23:31:16.353127 3698 raft_consensus.cc:3032] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:16.358042 3746 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:16.404795 1581 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:16.405273 1587 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:16.308727 3524 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.353082 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:16.421811 1628 leader_election.cc:334] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.898s (SENT)
I20260927 23:31:16.193212 3742 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.249s
I20260927 23:31:16.432811 1627 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:16.361900 3758 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:16.381704 3748 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:16.395921 3719 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:16.398473 3506 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 23:31:16.434660 3721 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 23:31:16.407346 3698 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:16.439065 3728 log.cc:924] Time spent T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260927 23:31:16.361449 3745 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.376236 3535 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.063s
I20260927 23:31:16.325435 3740 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20260927 23:31:16.488876 1588 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:16.368866 3134 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.438925 2286 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:16.413249 3689 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:16.513129 3689 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:16.513581 3689 raft_consensus.cc:686] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:16.513145 2901 consensus_meta.cc:287] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.517s user 0.006s sys 0.002s
I20260927 23:31:16.515089 3689 consensus_queue.cc:237] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:16.488482 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 55755.8us
I20260927 23:31:16.449065 3758 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:16.439843 3680 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.450960 3748 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:16.496691 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:16.496299 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.500743 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:16.476804 3707 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.511103 3134 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.511685 3766 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:16.406695 3754 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20260927 23:31:16.516920 1735 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:16.540094 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 202) took 1692 ms. Trace:
W20260927 23:31:16.567195 1628 leader_election.cc:334] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.542s (SENT)
I20260927 23:31:16.501259 3717 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.376623 3751 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.475692 3719 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.620002 2294 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 460) took 1780 ms. Trace:
I20260927 23:31:16.682706 3752 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:16.516583 3743 log.cc:924] Time spent T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20260927 23:31:16.694950 2293 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260927 23:31:16.699211 2293 pending_rounds.cc:76] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting all ops after (but not including) 5
I20260927 23:31:16.699453 2293 pending_rounds.cc:97] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260927 23:31:16.693713 2294 rpcz_store.cc:276] 0927 23:31:14.839669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.839806 (+ 137us) service_pool.cc:227] Handling call
0927 23:31:14.840141 (+ 335us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:15.358937 (+518796us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:15.358967 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:15.359450 (+ 483us) log.cc:842] Serialized 26 byte log entry
0927 23:31:15.424234 (+ 64784us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:15.460416 (+ 36182us) log.cc:842] Serialized 14 byte log entry
0927 23:31:15.460580 (+ 164us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:15.460655 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:15.908358 (+447703us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b1800271d80. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:16.538115 (+629757us) raft_consensus.cc:1671] finished
0927 23:31:16.619765 (+ 81650us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:16.619930 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1672,"prepare.queue_time_us":35871,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":150452,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":18891,"spinlock_wait_cycles":86635648,"thread_start_us":579733,"threads_started":2,"wal-append.queue_time_us":533875}
I20260927 23:31:16.699707 2293 raft_consensus.cc:2866] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:16.516597 3758 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:16.640631 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:16.713482 1660 rpcz_store.cc:276] 0927 23:31:14.847804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.096438 (+248634us) service_pool.cc:227] Handling call
0927 23:31:15.139098 (+ 42660us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:15.145374 (+ 6276us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:15.145414 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:15.161439 (+ 16025us) log.cc:842] Serialized 26 byte log entry
0927 23:31:15.397671 (+236232us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:15.446962 (+ 49291us) log.cc:842] Serialized 14 byte log entry
0927 23:31:15.447168 (+ 206us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:15.447239 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:16.539812 (+1092573us) raft_consensus.cc:1671] finished
0927 23:31:16.539838 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:16.540020 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":50338,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":30218,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10001,"spinlock_wait_cycles":6690560,"thread_start_us":235185,"threads_started":1,"wal-append.queue_time_us":598282}
I20260927 23:31:16.700008 3772 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.716805 3672 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.684397 3707 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:16.747499 3117 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260927 23:31:16.496981 3702 consensus_queue.cc:237] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.620366 3698 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.762879 3698 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:16.763535 3762 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
I20260927 23:31:16.761137 3711 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.765182 2288 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 431) took 1008 ms. Trace:
W20260927 23:31:16.766708 3725 consensus_meta.cc:287] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.630s user 0.000s sys 0.011s
I20260927 23:31:16.765704 2288 rpcz_store.cc:276] 0927 23:31:15.756775 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.763901 (+ 7126us) service_pool.cc:227] Handling call
0927 23:31:15.849519 (+ 85618us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:16.012584 (+163065us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:16.057748 (+ 45164us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:16.118024 (+ 60276us) log.cc:842] Serialized 26 byte log entry
0927 23:31:16.382808 (+264784us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:16.382863 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:16.382970 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:16.764901 (+381931us) raft_consensus.cc:1671] finished
0927 23:31:16.764928 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:16.765106 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":59984,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":298903,"threads_started":2,"wal-append.queue_time_us":325913}
I20260927 23:31:16.767100 3725 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.768532 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:16.769074 3725 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.694053 3752 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.769341 2289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51538 (request call id 432) took 1002 ms. Trace:
I20260927 23:31:16.769712 2289 rpcz_store.cc:276] 0927 23:31:15.766908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.812503 (+ 45595us) service_pool.cc:227] Handling call
0927 23:31:16.768377 (+955874us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:16.769283 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5152128}
I20260927 23:31:16.761631 3524 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.759474 3680 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:16.760347 3707 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.763139 3772 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.771817 3643 raft_consensus.cc:3032] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:16.772413 3705 consensus_meta.cc:287] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.838s user 0.008s sys 0.001s
I20260927 23:31:16.705046 3766 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.740298 3717 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:16.789179 3590 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.886s user 0.009s sys 0.001s
I20260927 23:31:16.789685 3590 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.791230 3717 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.792029 3590 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.788815 3752 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.713882 3758 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.805927 3758 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.798128 3745 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:16.800189 3748 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.807396 3751 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.797300 3768 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:16.808189 3768 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:16.808808 3768 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.809103 3748 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.809751 3768 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:16.811570 3772 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.793007 3705 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.813907 3705 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.796128 3707 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.819569 3766 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.801982 3746 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:16.850872 1551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.000s sys 0.010s
W20260927 23:31:16.852046 1551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.000s sys 0.012s
W20260927 23:31:16.852623 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 558182us
I20260927 23:31:16.854221 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.856838 1582 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:16.857911 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:16.539743 1580 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
W20260927 23:31:16.873432 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 306137us
W20260927 23:31:16.878626 3470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.672s user 0.000s sys 0.021s
W20260927 23:31:16.879215 3470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.680s user 0.000s sys 0.021s
I20260927 23:31:16.879243 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:16.880693 1736 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:16.880734 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1483) took 1555 ms. Trace:
W20260927 23:31:16.880920 1628 leader_election.cc:334] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.919s (SENT)
I20260927 23:31:16.882673 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:16.884239 2285 raft_consensus.cc:2396] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260927 23:31:16.881436 1655 rpcz_store.cc:276] 0927 23:31:15.324773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.831654 (+506881us) service_pool.cc:227] Handling call
0927 23:31:15.832084 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:15.832463 (+ 379us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:15.832497 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:15.833040 (+ 543us) log.cc:842] Serialized 26 byte log entry
0927 23:31:16.078901 (+245861us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:16.117745 (+ 38844us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:16.124628 (+ 6883us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:16.880029 (+755401us) raft_consensus.cc:1671] finished
0927 23:31:16.880084 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:16.880433 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15780,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":76,"thread_start_us":238723,"threads_started":1,"wal-append.queue_time_us":412555}
I20260927 23:31:16.887101 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:16.890969 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:16.891131 2287 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:16.891834 1655 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:16.881696 1628 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:16.888262 1660 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
W20260927 23:31:16.803162 3718 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.735s user 0.007s sys 0.003s
I20260927 23:31:16.786957 3643 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:16.908465 3742 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:16.912397 3760 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:16.913290 3761 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:16.914197 3393 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:16.915823 1702 leader_election.cc:334] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.959s (SENT)
I20260927 23:31:16.941195 1582 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:16.909890 3654 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:16.786063 3680 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.957597 3772 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:16.958302 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:16.960875 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 80075.9us
W20260927 23:31:16.962097 1628 leader_election.cc:334] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.744s (SENT)
I20260927 23:31:16.962723 3718 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.963037 3761 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:16.909976 3664 consensus_meta.cc:287] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.510s user 0.010s sys 0.001s
I20260927 23:31:16.962709 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:16.968092 3642 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:16.968472 3718 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.970620 1581 raft_consensus.cc:2379] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:16.971568 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:16.972407 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:16.973446 1733 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:16.973826 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:16.974668 1729 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:16.974926 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:16.975816 1728 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:16.976250 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:16.977030 1731 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:16.977140 3705 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:16.970634 3772 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.978045 3705 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:16.982370 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:16.983171 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:16.984239 3702 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:16.984479 1660 raft_consensus.cc:2354] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5.
I20260927 23:31:16.984828 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:16.985548 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:16.985668 1734 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:16.987809 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:16.988492 3680 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:16.990063 3760 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:16.991003 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.991395 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.992496 2286 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:16.992669 3760 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:16.992822 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.996656 1731 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:16.998296 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.015277 1588 raft_consensus.cc:2362] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:16.994134 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.970293 1703 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:17.016391 2293 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:17.052233 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.054129 2288 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:17.048202 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:17.048399 3654 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.056489 1731 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:17.058239 3654 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.048766 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:17.007989 1628 leader_election.cc:334] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.631s (SENT)
I20260927 23:31:17.016425 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.049345 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:17.052520 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:17.066100 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.066278 3393 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.067369 3643 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.068720 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:17.069661 2286 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 5
I20260927 23:31:17.069720 3705 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:17.070925 3761 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.071849 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:17.072121 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.011370 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1640) took 2522 ms. Trace:
I20260927 23:31:17.003319 1734 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:16.992542 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.070825 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.070834 1736 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:17.067119 2287 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:17.075927 1731 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:17.076515 1729 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:17.008852 1627 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.010345 1732 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:17.075307 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 58657.5us
I20260927 23:31:17.072765 1663 rpcz_store.cc:276] 0927 23:31:14.487856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.001770 (+513914us) service_pool.cc:227] Handling call
0927 23:31:15.002079 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:15.415974 (+413895us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.009920 (+1593946us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400447fd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:17.010284 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":926580096}
I20260927 23:31:17.026038 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:17.082785 1585 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:17.085500 1588 raft_consensus.cc:2354] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:17.066167 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.088936 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:17.089975 1663 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 7
W20260927 23:31:17.090524 3736 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
I20260927 23:31:17.090744 3654 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:17.091526 3439 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:17.093262 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.095309 1554 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:17.096117 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 87607us
I20260927 23:31:17.101243 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.106665 3503 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:17.107151 3503 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:17.107002 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.107775 3503 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:17.110150 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.111999 3117 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:17.112475 2293 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:17.112649 1728 raft_consensus.cc:1261] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:17.112932 2285 raft_consensus.cc:2354] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7.
I20260927 23:31:17.110671 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.111304 1587 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:17.114120 1582 raft_consensus.cc:2354] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5.
I20260927 23:31:17.115072 1583 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:17.115783 3500 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:17.116226 3500 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:17.116779 3500 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:17.116144 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.119484 3704 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:17.119885 3704 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:17.120994 3654 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:17.121650 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.126879 1554 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:17.127490 1583 raft_consensus.cc:2354] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5.
W20260927 23:31:17.128181 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 120313us
W20260927 23:31:17.129004 1628 leader_election.cc:334] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.047s (SENT)
I20260927 23:31:17.121109 3704 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.087965 3772 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.126492 1660 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:17.127036 1655 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:17.125563 3719 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:17.117452 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 147574us
I20260927 23:31:17.107187 1627 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.130124 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.162806 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.097219 3393 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.129227 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.147612 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:17.156213 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:17.157757 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 82029.2us
I20260927 23:31:17.157946 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1494) took 1002 ms. Trace:
I20260927 23:31:17.173313 1655 rpcz_store.cc:276] 0927 23:31:16.148706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:16.888121 (+739415us) service_pool.cc:227] Handling call
0927 23:31:17.151298 (+263177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:17.129876 3654 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.158529 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:16.999481 3742 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:17.164395 1552 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46522 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 5 similar messages]
W20260927 23:31:17.135545 3750 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.662s user 0.002s sys 0.000s
I20260927 23:31:17.176283 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.177115 1663 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:17.179298 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:17.180940 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 73883.9us
I20260927 23:31:17.129643 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.181247 3696 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:17.183686 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:17.184643 3712 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:17.185045 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:17.186741 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 271184us
I20260927 23:31:17.189463 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.188079 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.202485 1579 raft_consensus.cc:2354] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
W20260927 23:31:17.195070 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 66137.2us
I20260927 23:31:17.201970 1587 raft_consensus.cc:2354] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:17.202535 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.183338 3704 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:17.206228 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:17.210023 3492 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:17.210588 3492 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:17.211555 3492 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.212569 3793 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:17.213393 3439 consensus_queue.cc:1046] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:17.213192 3793 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.214773 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.214785 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:17.216398 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:17.191589 3712 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:17.217007 3793 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:17.217193 3712 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:17.217685 1655 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:17.219717 2286 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:17.219897 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1649) took 2100 ms. Trace:
I20260927 23:31:17.207110 1588 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:17.220306 1656 rpcz_store.cc:276] 0927 23:31:15.119434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.400367 (+280933us) service_pool.cc:227] Handling call
0927 23:31:15.400677 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:15.757472 (+356795us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.219480 (+1462008us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54002dd7d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:17.219745 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684064512}
I20260927 23:31:17.229861 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:17.230422 1736 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:17.230892 1733 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:17.234695 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:17.234879 3500 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.236219 3664 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:17.227560 3117 raft_consensus.cc:3032] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:17.236817 3117 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:17.236909 3664 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.237437 3500 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:17.237946 3117 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.239221 3664 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.239461 3117 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:17.240216 1657 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:17.240413 1628 leader_election.cc:334] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.779s (SENT)
I20260927 23:31:17.240923 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1481) took 1916 ms. Trace:
I20260927 23:31:17.241297 1657 rpcz_store.cc:276] 0927 23:31:15.324136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.739920 (+415784us) service_pool.cc:227] Handling call
0927 23:31:17.239551 (+1499631us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:17.239835 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:17.240664 (+ 829us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.240870 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3257600}
I20260927 23:31:17.242594 3758 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:17.245378 2258 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.246244 3793 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:17.246685 3793 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:17.247109 3793 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:17.247227 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1633) took 2766 ms. Trace:
I20260927 23:31:17.248034 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1630) took 2776 ms. Trace:
I20260927 23:31:17.247653 1659 rpcz_store.cc:276] 0927 23:31:14.480526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.480653 (+ 127us) service_pool.cc:227] Handling call
0927 23:31:14.547355 (+ 66702us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:15.914846 (+1367491us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.246790 (+1331944us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54003f4154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:17.247053 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2995713792}
I20260927 23:31:17.248507 1662 rpcz_store.cc:276] 0927 23:31:14.471448 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.472033 (+ 585us) service_pool.cc:227] Handling call
0927 23:31:14.495402 (+ 23369us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:14.843333 (+347931us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:14.988210 (+144877us) log.cc:842] Serialized 14 byte log entry
0927 23:31:15.207688 (+219478us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:15.207750 (+ 62us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:15.207801 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.247645 (+2039844us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b54002f30d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:17.247700 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:17.247880 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":170580,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":19340,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5806,"spinlock_wait_cycles":799596928,"thread_start_us":300406,"threads_started":2,"wal-append.queue_time_us":257378,"wal-append.run_cpu_time_us":6809,"wal-append.run_wall_time_us":1444062}
I20260927 23:31:17.235562 1728 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:17.246785 2258 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.205892 3696 raft_consensus.cc:686] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:17.196223 3742 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.195485 3772 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:17.256625 1583 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:17.257578 1663 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:17.259984 1662 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:17.260643 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1486) took 1794 ms. Trace:
I20260927 23:31:17.260946 1662 rpcz_store.cc:276] 0927 23:31:15.466000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.258896 (+1792896us) service_pool.cc:227] Handling call
0927 23:31:17.259542 (+ 646us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:17.260364 (+ 822us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.260564 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48128}
I20260927 23:31:17.262416 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1292) took 1187 ms. Trace:
I20260927 23:31:17.262792 1730 rpcz_store.cc:276] 0927 23:31:16.074971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:16.075149 (+ 178us) service_pool.cc:227] Handling call
0927 23:31:17.262036 (+1186887us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b54002bdc54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:17.262202 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45385600}
I20260927 23:31:17.226306 3439 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:17.264830 1662 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:17.265429 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.267032 3470 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:17.266870 3696 consensus_queue.cc:237] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.267817 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1492) took 1185 ms. Trace:
I20260927 23:31:17.267661 3470 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.268262 3792 raft_consensus.cc:2781] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:17.268945 3792 raft_consensus.cc:686] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:17.268258 1663 rpcz_store.cc:276] 0927 23:31:16.081957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.266594 (+1184637us) service_pool.cc:227] Handling call
0927 23:31:17.266992 (+ 398us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:17.267451 (+ 459us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:17.267480 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:17.267524 (+ 44us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:17.267577 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.267638 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:17.267752 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:17.271303 3516 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:17.276933 3510 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.277663 3510 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:17.283592 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 56921.6us
I20260927 23:31:17.285524 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:17.287045 3745 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:17.283664 3666 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 23:31:17.287083 3796 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260927 23:31:17.265836 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:17.290504 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 76268.1us
I20260927 23:31:17.294796 1656 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
W20260927 23:31:17.299515 1628 leader_election.cc:334] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.852s (SENT)
I20260927 23:31:17.270184 3470 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:17.303470 1655 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:17.293412 3745 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:17.270002 3792 consensus_queue.cc:237] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.316448 3502 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
W20260927 23:31:17.322378 3756 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.762s user 0.004s sys 0.025s
W20260927 23:31:17.322909 3756 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.888s user 0.004s sys 0.026s
I20260927 23:31:17.187667 1552 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46522 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1306)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1282)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1318)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1317)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1283)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1288)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1307)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1373)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1281)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1319)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1299)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1311)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1298)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 293)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1301)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1305)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1310)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1302)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1371)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1308)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1372)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1276)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1315)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1271)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1320)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1275)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1297)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1314)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1270)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1277)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1313)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1370)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 291)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1303)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1309)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1316)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1374)
I20260927 23:31:17.293234 3666 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:17.280143 3439 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.301781 3758 consensus_queue.cc:1035] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260927 23:31:17.278306 3510 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:17.338107 3740 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.014s sys 0.024s
I20260927 23:31:17.362756 1662 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:17.339391 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:17.343149 3666 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:17.399220 3740 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.888s user 0.014s sys 0.025s
W20260927 23:31:17.742693 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 455281us
I20260927 23:31:17.797777 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 209) took 1291 ms. Trace:
W20260927 23:31:17.800205 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 506325us
I20260927 23:31:17.804020 1655 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:17.738950 3690 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.823s user 0.019s sys 0.018s
W20260927 23:31:17.741459 3790 log.cc:924] Time spent T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.673s user 0.003s sys 0.000s
I20260927 23:31:17.488058 3510 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:17.802539 3525 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:17.819943 1702 leader_election.cc:334] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.775s (SENT)
I20260927 23:31:17.525688 3439 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:17.813356 3690 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.897s user 0.019s sys 0.022s
I20260927 23:31:17.824968 1702 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:17.829012 3745 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.831749 3470 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 23:31:17.833599 3470 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:17.643357 3793 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:17.834260 3470 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:17.836803 3470 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:17.837199 1733 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:17.840344 1702 leader_election.cc:334] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.522s (SENT)
W20260927 23:31:17.842912 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 44903.9us
I20260927 23:31:17.827782 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:17.860843 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 44269.1us
W20260927 23:31:17.868448 3639 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.847s user 0.011s sys 0.018s
W20260927 23:31:17.868844 3639 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.877s user 0.011s sys 0.019s
W20260927 23:31:17.873173 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 49311.9us
I20260927 23:31:17.876210 1702 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:17.933354 3756 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.008s sys 0.026s
W20260927 23:31:17.978194 3503 consensus_meta.cc:287] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.870s user 0.013s sys 0.002s
W20260927 23:31:18.009315 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 164352us
I20260927 23:31:18.005865 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:17.976342 3492 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.012s sys 0.019s
I20260927 23:31:17.936698 3761 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:18.005311 3756 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.008s sys 0.026s
W20260927 23:31:18.025170 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 224709us
W20260927 23:31:18.017201 3642 consensus_meta.cc:287] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.048s user 0.015s sys 0.000s
W20260927 23:31:18.020788 3492 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.012s sys 0.019s
I20260927 23:31:18.015003 3696 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.028813 1629 leader_election.cc:398] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7.
I20260927 23:31:18.033083 1629 leader_election.cc:401] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260927 23:31:18.025367 3666 consensus_meta.cc:287] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.003s
I20260927 23:31:18.038620 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:18.039073 3712 consensus_meta.cc:287] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.821s user 0.008s sys 0.003s
W20260927 23:31:18.040019 3117 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.800s user 0.009s sys 0.004s
I20260927 23:31:18.033749 3642 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:18.029620 3770 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.514s user 0.008s sys 0.013s
W20260927 23:31:18.059538 3795 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 1.089s user 0.009s sys 0.011s
I20260927 23:31:18.061333 3754 raft_consensus.cc:3032] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:18.062593 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.099113 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1289) took 1228 ms. Trace:
I20260927 23:31:18.063793 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.066497 3763 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.027315 3781 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.068892 3117 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.067101 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.067153 3464 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:18.070477 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 55577us
I20260927 23:31:18.110809 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1317) took 1008 ms. Trace:
I20260927 23:31:18.092773 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1282) took 1226 ms. Trace:
I20260927 23:31:18.098592 3642 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7.
I20260927 23:31:18.099453 1587 rpcz_store.cc:276] 0927 23:31:16.870480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.062261 (+1191781us) service_pool.cc:227] Handling call
0927 23:31:18.099056 (+ 36795us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.097707 3770 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.582s user 0.008s sys 0.014s
I20260927 23:31:18.101415 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1293) took 1212 ms. Trace:
I20260927 23:31:18.068979 3809 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:18.019194 3503 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:18.093257 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 253011us
I20260927 23:31:18.111249 1588 rpcz_store.cc:276] 0927 23:31:17.102875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.066805 (+963930us) service_pool.cc:227] Handling call
0927 23:31:18.110728 (+ 43923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.110704 1629 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:18.111950 1579 rpcz_store.cc:276] 0927 23:31:16.866878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.038253 (+1171375us) service_pool.cc:227] Handling call
0927 23:31:18.092673 (+ 54420us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.051487 3666 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.052537 3712 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.100191 3793 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.071008 3470 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.067870 3696 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.049024 3761 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.115266 1586 rpcz_store.cc:276] 0927 23:31:16.889157 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.063515 (+1174358us) service_pool.cc:227] Handling call
0927 23:31:18.101360 (+ 37845us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.100406 3754 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:18.113615 3707 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.042702 3772 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.069s
W20260927 23:31:18.125554 1628 leader_election.cc:334] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.734s (SENT)
I20260927 23:31:18.126523 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.128108 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:18.127857 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.128737 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1288) took 1258 ms. Trace:
I20260927 23:31:18.129742 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.117547 3745 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.130286 1579 rpcz_store.cc:276] 0927 23:31:16.870051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.120432 (+1250381us) service_pool.cc:227] Handling call
0927 23:31:18.128672 (+ 8240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.130972 3745 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.131307 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.132179 1655 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:18.132565 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.133620 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1373) took 1028 ms. Trace:
I20260927 23:31:18.133994 1587 rpcz_store.cc:276] 0927 23:31:17.105105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.132263 (+1027158us) service_pool.cc:227] Handling call
0927 23:31:18.133539 (+ 1276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.135764 1586 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:18.136457 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1307) took 1169 ms. Trace:
I20260927 23:31:18.136873 1586 rpcz_store.cc:276] 0927 23:31:17.017240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.123673 (+1106433us) service_pool.cc:227] Handling call
0927 23:31:18.136386 (+ 12713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.118825 3751 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.125s
I20260927 23:31:18.139213 1588 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 2.
I20260927 23:31:18.139740 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1283) took 1272 ms. Trace:
I20260927 23:31:18.138031 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.144011 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1299) took 1252 ms. Trace:
I20260927 23:31:18.144470 1586 rpcz_store.cc:276] 0927 23:31:16.891853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.137689 (+1245836us) service_pool.cc:227] Handling call
0927 23:31:18.143939 (+ 6250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.145432 1702 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.877s (SENT)
I20260927 23:31:18.140031 1588 rpcz_store.cc:276] 0927 23:31:16.867314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.120054 (+1252740us) service_pool.cc:227] Handling call
0927 23:31:18.139689 (+ 19635us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.134424 2257 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.012s sys 0.005s
W20260927 23:31:18.146109 2257 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.707s user 0.012s sys 0.005s
W20260927 23:31:18.146396 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 795716us
I20260927 23:31:18.123667 3464 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:18.148759 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.150712 3666 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.153052 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.172359 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1319) took 1068 ms. Trace:
I20260927 23:31:18.156127 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:18.156080 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.119367 3781 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.124513 3763 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.126199 3117 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.174413 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1311) took 1073 ms. Trace:
W20260927 23:31:18.193420 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1269) took 1686 ms (client timeout 1631 ms). Trace:
I20260927 23:31:18.156351 3492 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:18.217166 1588 rpcz_store.cc:276] 0927 23:31:17.100793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.146083 (+1045290us) service_pool.cc:227] Handling call
0927 23:31:18.174257 (+ 28174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.174127 1587 rpcz_store.cc:276] 0927 23:31:17.103505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.134644 (+1031139us) service_pool.cc:227] Handling call
0927 23:31:18.172277 (+ 37633us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.216708 1586 raft_consensus.cc:2362] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:18.224941 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1312) took 1123 ms. Trace:
I20260927 23:31:18.225203 1586 rpcz_store.cc:276] 0927 23:31:17.101187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.145740 (+1044553us) service_pool.cc:227] Handling call
0927 23:31:18.224883 (+ 79143us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.217837 1580 rpcz_store.cc:269] 0927 23:31:16.420228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.070012 (+1649784us) service_pool.cc:214] Skipping call since client already timed out
0927 23:31:18.106557 (+ 36545us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 23:31:18.202273 1658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1482) took 2775 ms (client timeout 2441 ms). Trace:
W20260927 23:31:18.176133 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 65508.8us
I20260927 23:31:18.148382 3642 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:18.226487 3713 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:18.226943 3464 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.227092 3713 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.227946 3781 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.226533 3642 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:18.228830 3642 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:18.229269 3713 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.226207 1629 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:18.231704 3705 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:18.232250 3705 raft_consensus.cc:686] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:18.233173 3755 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:18.233880 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:18.234299 3816 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260927 23:31:18.153540 3712 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:18.234911 1587 raft_consensus.cc:2362] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:18.234167 3755 raft_consensus.cc:3032] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:18.235415 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1298) took 1344 ms. Trace:
I20260927 23:31:18.235459 3755 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:18.235989 2286 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 23:31:18.236676 1551 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:18.236589 3755 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.237871 3755 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:18.235894 1587 rpcz_store.cc:276] 0927 23:31:16.891410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.233601 (+1342191us) service_pool.cc:227] Handling call
0927 23:31:18.235369 (+ 1768us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.225854 1658 rpcz_store.cc:269] 0927 23:31:15.324473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.069630 (+2745157us) service_pool.cc:214] Skipping call since client already timed out
0927 23:31:18.100265 (+ 30635us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 23:31:18.186112 3814 log.cc:924] Time spent T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260927 23:31:18.233286 3705 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:18.241320 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 118292us
I20260927 23:31:18.237833 3475 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:18.217790 3745 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:18.251667 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.252012 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.254701 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.255033 1587 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:18.255264 1586 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:18.251296 1579 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:18.256615 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 293) took 1394 ms. Trace:
I20260927 23:31:18.255548 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1301) took 1362 ms. Trace:
I20260927 23:31:18.257081 1586 rpcz_store.cc:276] 0927 23:31:16.862140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.240165 (+1378025us) service_pool.cc:227] Handling call
0927 23:31:18.256561 (+ 16396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.257190 1587 rpcz_store.cc:276] 0927 23:31:16.892677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.239675 (+1346998us) service_pool.cc:227] Handling call
0927 23:31:18.255473 (+ 15798us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.251808 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.255417 1588 raft_consensus.cc:2362] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:18.258971 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1287) took 1389 ms. Trace:
I20260927 23:31:18.259330 1588 rpcz_store.cc:276] 0927 23:31:16.869583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.240084 (+1370501us) service_pool.cc:227] Handling call
0927 23:31:18.258919 (+ 18835us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29568}
I20260927 23:31:18.252699 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:18.242369 3712 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:18.186347 3470 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.258215 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.257409 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:18.272521 2286 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:18.272684 3696 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:18.264671 3806 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
W20260927 23:31:18.270668 1628 leader_election.cc:334] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.120s (SENT)
W20260927 23:31:18.271363 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 136965us
I20260927 23:31:18.278362 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.280663 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1305) took 1386 ms. Trace:
I20260927 23:31:18.281188 1580 rpcz_store.cc:276] 0927 23:31:16.894176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.239824 (+1345648us) service_pool.cc:227] Handling call
0927 23:31:18.280582 (+ 40758us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160128}
I20260927 23:31:18.271919 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1655) took 1181 ms. Trace:
I20260927 23:31:18.252803 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:18.304621 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1310) took 1204 ms. Trace:
I20260927 23:31:18.328507 1586 rpcz_store.cc:276] 0927 23:31:17.100446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.257948 (+1157502us) service_pool.cc:227] Handling call
0927 23:31:18.304544 (+ 46596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.174902 3707 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.295096 1662 rpcz_store.cc:276] 0927 23:31:17.090867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.419411 (+328544us) service_pool.cc:227] Handling call
0927 23:31:17.798994 (+379583us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:17.799299 (+ 305us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:17.799850 (+ 551us) log.cc:842] Serialized 14 byte log entry
0927 23:31:18.271705 (+471855us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:18.271738 (+ 33us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:18.271770 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.271808 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:18.271880 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":179,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":10995,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":441887,"threads_started":1}
I20260927 23:31:18.285351 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.284037 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:18.329746 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 44790.8us
W20260927 23:31:18.330121 3757 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.006s sys 0.020s
I20260927 23:31:18.333845 2257 leader_election.cc:398] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5.
I20260927 23:31:18.337116 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1371) took 1232 ms. Trace:
I20260927 23:31:18.337759 2257 leader_election.cc:401] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:18.337808 1588 rpcz_store.cc:276] 0927 23:31:17.104484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.260000 (+1155516us) service_pool.cc:227] Handling call
0927 23:31:18.337045 (+ 77045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.333364 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.341204 1551 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:18.274993 3745 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:18.270413 3713 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:18.284827 3812 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260927 23:31:18.340476 1580 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:18.352627 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1308) took 1279 ms. Trace:
I20260927 23:31:18.345959 1658 raft_consensus.cc:2379] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:18.353140 1580 rpcz_store.cc:276] 0927 23:31:17.075482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.282255 (+1206773us) service_pool.cc:227] Handling call
0927 23:31:18.352557 (+ 70302us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.350651 1702 leader_election.cc:334] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.212s (SENT)
I20260927 23:31:18.349309 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.355263 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1372) took 1250 ms. Trace:
I20260927 23:31:18.355670 1586 rpcz_store.cc:276] 0927 23:31:17.104795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.329256 (+1224461us) service_pool.cc:227] Handling call
0927 23:31:18.355195 (+ 25939us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.338167 3757 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.006s sys 0.020s
W20260927 23:31:18.342710 3525 consensus_meta.cc:287] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.516s user 0.004s sys 0.007s
I20260927 23:31:18.361696 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:18.360387 3810 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20260927 23:31:18.360849 2257 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:18.361292 3712 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:18.361512 3134 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5.
I20260927 23:31:18.341379 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:18.361806 3654 consensus_queue.cc:1046] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:31:18.362587 3591 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:18.363130 3687 consensus_queue.cc:1046] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
W20260927 23:31:18.364535 3803 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260927 23:31:18.372562 1588 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:18.374471 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 103826us
I20260927 23:31:18.375849 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.343528 3503 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:18.350698 3492 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.383255 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.393699 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.395658 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1315) took 1293 ms. Trace:
W20260927 23:31:18.396955 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64329.6us
I20260927 23:31:18.397284 1580 rpcz_store.cc:276] 0927 23:31:17.102218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.354139 (+1251921us) service_pool.cc:227] Handling call
0927 23:31:18.395593 (+ 41454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.398851 3825 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.399663 3825 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.402370 3763 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.402750 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:18.402985 3492 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.403571 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:18.404386 1732 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:18.405308 3492 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.405455 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:18.405781 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:18.406199 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.406404 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.406687 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:18.407038 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1320) took 1303 ms. Trace:
I20260927 23:31:18.407545 3470 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.407946 3825 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:18.408661 3805 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20260927 23:31:18.388813 3666 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:18.409395 3666 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:18.409574 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.375135 3713 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.409797 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1491) took 2685 ms. Trace:
I20260927 23:31:18.410349 1662 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:18.411360 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.411576 3713 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.410427 1654 rpcz_store.cc:276] 0927 23:31:15.724288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.258840 (+1534552us) service_pool.cc:227] Handling call
0927 23:31:17.259502 (+ 662us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:17.260078 (+ 576us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:17.260121 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:17.260651 (+ 530us) log.cc:842] Serialized 26 byte log entry
0927 23:31:17.399760 (+139109us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:17.404990 (+ 5230us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.405077 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:18.409578 (+1004501us) raft_consensus.cc:1671] finished
0927 23:31:18.409600 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:18.409740 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4961,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":16601,"spinlock_wait_cycles":22656,"thread_start_us":112104,"threads_started":1,"wal-append.queue_time_us":884925}
I20260927 23:31:18.409770 3741 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.412038 1662 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:18.403505 1729 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:18.413115 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:18.407105 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1271) took 1956 ms. Trace:
I20260927 23:31:18.407426 1580 rpcz_store.cc:276] 0927 23:31:17.103796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.398007 (+1294211us) service_pool.cc:227] Handling call
0927 23:31:18.406984 (+ 8977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.414863 1736 raft_consensus.cc:2379] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:18.260686 3754 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.407327 1728 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:18.389262 2288 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:18.413626 1658 raft_consensus.cc:2396] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:18.414904 1586 rpcz_store.cc:276] 0927 23:31:16.451059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.379532 (+1928473us) service_pool.cc:227] Handling call
0927 23:31:18.407055 (+ 27523us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.403143 1733 raft_consensus.cc:1261] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:18.392246 3696 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.742420 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.505169 3761 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:18.743379 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1297) took 1852 ms. Trace:
I20260927 23:31:18.442051 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:18.444108 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:18.744812 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1657) took 1653 ms. Trace:
I20260927 23:31:18.480731 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.409807 3666 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:18.758783 1732 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:18.759544 1587 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:18.426930 3741 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.521692 1728 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:18.488611 1662 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:18.743778 1586 rpcz_store.cc:276] 0927 23:31:16.891073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.742107 (+1851034us) service_pool.cc:227] Handling call
0927 23:31:18.743331 (+ 1224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.693032 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 274) took 1406 ms. Trace:
I20260927 23:31:18.768316 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1661) took 1649 ms. Trace:
I20260927 23:31:18.768896 1662 rpcz_store.cc:276] 0927 23:31:17.119238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.422580 (+1303342us) service_pool.cc:227] Handling call
0927 23:31:18.488369 (+ 65789us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:18.768009 (+279640us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.768251 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.525538 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 164436us
I20260927 23:31:18.479545 1736 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:18.761628 1580 raft_consensus.cc:2354] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4.
I20260927 23:31:18.417001 2289 raft_consensus.cc:2354] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:18.756356 1734 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:18.499845 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 74595.3us
I20260927 23:31:18.568280 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:18.781420 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1306) took 1863 ms (client timeout 1792 ms). Trace:
W20260927 23:31:18.781955 1585 rpcz_store.cc:269] 0927 23:31:16.941718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.498880 (+1557162us) service_pool.cc:227] Handling call
0927 23:31:18.781330 (+282450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.409870 3654 consensus_queue.cc:1225] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260927 23:31:18.703775 3828 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.100s user 0.002s sys 0.000s
W20260927 23:31:18.560370 1702 leader_election.cc:334] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.246s (SENT)
W20260927 23:31:18.784245 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1281) took 1926 ms (client timeout 1995 ms). Trace:
W20260927 23:31:18.848511 1579 rpcz_store.cc:269] 0927 23:31:16.858288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.131054 (+1272766us) service_pool.cc:227] Handling call
0927 23:31:18.784047 (+652993us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003093d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:18.784183 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":261129984}
W20260927 23:31:18.849213 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1275) took 1993 ms (client timeout 2576 ms). Trace:
W20260927 23:31:18.849861 1580 rpcz_store.cc:269] 0927 23:31:16.855974 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.418517 (+1562543us) service_pool.cc:227] Handling call
0927 23:31:18.849133 (+430616us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.832125 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1267) took 2373 ms (client timeout 1744 ms). Trace:
I20260927 23:31:18.416997 3776 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:18.417197 3117 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.856022 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:18.864530 1728 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:18.856887 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:18.506706 3822 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.745332 1656 rpcz_store.cc:276] 0927 23:31:17.091650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.455204 (+363554us) service_pool.cc:227] Handling call
0927 23:31:17.800748 (+345544us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:17.984998 (+184250us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:17.985061 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:18.014296 (+ 29235us) log.cc:842] Serialized 26 byte log entry
0927 23:31:18.338811 (+324515us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:18.339087 (+ 276us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.339635 (+ 548us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:18.656974 (+317339us) raft_consensus.cc:1671] finished
0927 23:31:18.657004 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:18.744760 (+ 87756us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158864,"raft.queue_time_us":194036,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4946,"thread_start_us":291251,"threads_started":1,"wal-append.queue_time_us":611334}
I20260927 23:31:18.861459 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.866624 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.867228 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1314) took 1765 ms. Trace:
I20260927 23:31:18.492993 1654 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:18.478775 3772 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.431610 3470 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.881517 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:18.783999 3807 log.cc:924] Time spent T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.770s user 0.003s sys 0.002s
I20260927 23:31:18.881881 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:18.882647 1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1270) took 2434 ms (client timeout 3208 ms). Trace:
I20260927 23:31:18.604776 3665 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.867705 1586 rpcz_store.cc:276] 0927 23:31:17.101908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.766541 (+1664633us) service_pool.cc:227] Handling call
0927 23:31:18.867157 (+100616us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.883311 3665 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.884160 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.884828 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1370) took 1780 ms. Trace:
I20260927 23:31:18.885102 1586 rpcz_store.cc:276] 0927 23:31:17.104135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.883909 (+1779774us) service_pool.cc:227] Handling call
0927 23:31:18.884755 (+ 846us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.852051 1581 rpcz_store.cc:269] 0927 23:31:16.405236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:16.407487 (+ 2251us) service_pool.cc:227] Handling call
0927 23:31:16.957827 (+550340us) spinlock_profiling.cc:243] Waited 47.4 ms on lock 0x7b5800001898. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:18.778016 (+1820189us) spinlock_profiling.cc:243] Waited 6.2 ms on lock 0x7b5400331dd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:18.778191 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123357312}
I20260927 23:31:18.885950 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.874711 3822 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.889408 1734 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:18.891855 2258 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:18.881273 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.881045 3772 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.424890 3502 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.900128 1586 raft_consensus.cc:3037] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:18.900256 3502 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.901298 3772 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.881908 3470 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.766685 1730 rpcz_store.cc:276] 0927 23:31:17.287073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.287879 (+ 806us) service_pool.cc:227] Handling call
0927 23:31:17.303451 (+ 15572us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:17.794894 (+491443us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:17.794966 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:17.872377 (+ 77411us) log.cc:842] Serialized 26 byte log entry
0927 23:31:18.330937 (+458560us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:18.330975 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.331036 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:18.692728 (+361692us) raft_consensus.cc:1671] finished
0927 23:31:18.692755 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:18.692934 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56009,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":39159,"thread_start_us":410822,"threads_started":2,"wal-append.queue_time_us":636672}
I20260927 23:31:18.904681 1658 raft_consensus.cc:1261] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:18.744395 3707 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.886171 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:18.883046 1585 rpcz_store.cc:269] 0927 23:31:16.448105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.845309 (+2397204us) service_pool.cc:227] Handling call
0927 23:31:18.882597 (+ 37288us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:18.886705 1628 leader_election.cc:334] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.716s (SENT)
I20260927 23:31:18.882668 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.889992 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.567745 3713 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:18.893090 1579 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:18.894923 1728 raft_consensus.cc:1261] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:18.913475 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:18.897114 3757 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.897598 3830 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.920178 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.920117 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:18.920900 1581 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:18.897477 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.898672 3643 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:17.801388 1663 rpcz_store.cc:276] 0927 23:31:16.506184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.271401 (+765217us) service_pool.cc:227] Handling call
0927 23:31:17.271705 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:17.272220 (+ 515us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:17.272938 (+ 718us) log.cc:842] Serialized 14 byte log entry
0927 23:31:17.797192 (+524254us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:17.797248 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:17.797289 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:17.797356 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:17.797498 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99190,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":337880,"thread_start_us":188913,"threads_started":1}
I20260927 23:31:18.862780 3753 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260927 23:31:18.922793 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.899605 3822 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:18.893821 1554 leader_election.cc:398] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:18.923724 1554 leader_election.cc:401] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260927 23:31:18.893499 1700 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:18.904060 1588 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:18.925246 1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1276) took 2068 ms (client timeout 2447 ms). Trace:
W20260927 23:31:18.925771 1588 rpcz_store.cc:269] 0927 23:31:16.856604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.346359 (+1489755us) service_pool.cc:227] Handling call
0927 23:31:18.925181 (+578822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.418546 3642 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.903153 1733 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:18.897127 3665 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.908627 1627 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:18.825798 3755 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.109s
I20260927 23:31:18.911751 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1660) took 1793 ms. Trace:
I20260927 23:31:18.912919 1656 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:18.910387 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1313) took 1808 ms. Trace:
I20260927 23:31:18.867775 3696 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.934015 1580 rpcz_store.cc:276] 0927 23:31:17.101524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.865862 (+1764338us) service_pool.cc:227] Handling call
0927 23:31:18.910303 (+ 44441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.602448 3811 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.406s
I20260927 23:31:18.425726 3591 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:18.930891 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 75041.1us
I20260927 23:31:18.907492 3781 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:18.904732 3464 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:18.935189 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.914500 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:18.924163 3687 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.018s sys 0.041s
I20260927 23:31:18.918473 3134 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:18.917061 3758 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:18.940821 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.913475 3502 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:18.941174 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.942646 1662 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:18.891443 3776 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:18.940446 3712 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.010s sys 0.042s
W20260927 23:31:18.940332 3687 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.526s user 0.018s sys 0.041s
I20260927 23:31:18.950207 1655 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
W20260927 23:31:18.952116 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 466770us
I20260927 23:31:18.952971 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:18.953594 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.937099 3755 consensus_queue.cc:1225] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 0ebfcbb9c0574725a9df6ed2bf33ff47 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 23:31:18.938544 3492 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:18.936959 3707 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:18.955346 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.938000 3770 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:18.963485 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.975672 1588 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 2.
I20260927 23:31:18.946018 2288 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:18.965227 1700 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:18.966876 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:18.935926 1654 rpcz_store.cc:276] 0927 23:31:17.118856 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.414532 (+1295676us) service_pool.cc:227] Handling call
0927 23:31:18.424594 (+ 10062us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:18.879705 (+455111us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.908718 (+ 29013us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.984575 3822 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:18.984797 1585 raft_consensus.cc:2396] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 23:31:18.984951 3822 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:18.985411 3822 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:18.969175 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 82521.6us
I20260927 23:31:18.922108 3713 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:18.943094 2285 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.010s sys 0.060s
I20260927 23:31:18.971539 3707 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:18.983314 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:18.986202 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 52700.4us
I20260927 23:31:18.922441 3757 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.960109 3712 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:18.989708 1580 raft_consensus.cc:2379] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 3: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:18.968081 3492 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:18.998857 3492 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:19.000552 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.002581 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.003139 3591 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:19.003629 1583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1318) took 1900 ms (client timeout 1628 ms). Trace:
W20260927 23:31:19.004051 1583 rpcz_store.cc:269] 0927 23:31:17.103218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.003444 (+1900226us) service_pool.cc:214] Skipping call since client already timed out
0927 23:31:19.003547 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 23:31:19.004343 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:19.007530 1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1291) took 3087 ms (client timeout 2198 ms). Trace:
W20260927 23:31:19.007956 1735 rpcz_store.cc:269] 0927 23:31:15.920240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.926926 (+ 6686us) service_pool.cc:227] Handling call
0927 23:31:19.007207 (+3080281us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54003c4454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:19.007342 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1207305984}
I20260927 23:31:19.009831 3834 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:19.010807 3834 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:19.011121 3834 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:19.012012 3834 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.013438 3834 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:18.944150 3781 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.990515 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 291) took 2576 ms. Trace:
I20260927 23:31:19.027259 1585 rpcz_store.cc:276] 0927 23:31:16.410261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.985437 (+2575176us) service_pool.cc:227] Handling call
0927 23:31:18.985811 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:18.985976 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:18.986000 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:18.986056 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:18.986087 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.986139 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:18.986230 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.995925 3757 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:18.996739 3712 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:18.998174 1580 raft_consensus.cc:1261] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:18.989487 2285 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.581s user 0.010s sys 0.060s
I20260927 23:31:18.947022 3464 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:18.994216 1628 leader_election.cc:334] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.769s (SENT)
I20260927 23:31:19.033679 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:19.060339 1663 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:19.034687 1581 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:19.060829 2293 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:18.948993 3830 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.058686 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 290) took 2626 ms. Trace:
W20260927 23:31:19.063191 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 79949.7us
I20260927 23:31:19.058784 1579 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
W20260927 23:31:19.063894 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 226730us
W20260927 23:31:19.064137 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1277) took 2207 ms (client timeout 2854 ms). Trace:
W20260927 23:31:19.064476 1579 rpcz_store.cc:269] 0927 23:31:16.856943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.861304 (+2004361us) service_pool.cc:227] Handling call
0927 23:31:19.064089 (+202785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.059460 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1303) took 2140 ms. Trace:
I20260927 23:31:19.060714 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1309) took 1973 ms. Trace:
I20260927 23:31:19.065397 1580 rpcz_store.cc:276] 0927 23:31:16.893476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.993401 (+2099925us) service_pool.cc:227] Handling call
0927 23:31:18.997832 (+ 4431us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.033839 (+ 36007us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.034022 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:18.972203 3758 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:18.984954 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:18.993543 3707 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:19.069414 2257 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:19.074628 1579 raft_consensus.cc:1226] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:19.065616 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 131492us
I20260927 23:31:19.063486 1588 rpcz_store.cc:276] 0927 23:31:16.408712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.984221 (+2575509us) service_pool.cc:227] Handling call
0927 23:31:19.034500 (+ 50279us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.034690 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:19.034713 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:19.034765 (+ 52us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:19.034798 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.034850 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.034941 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:19.079998 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 85786.1us
I20260927 23:31:19.082124 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1376) took 1653 ms. Trace:
I20260927 23:31:19.082535 1579 rpcz_store.cc:276] 0927 23:31:17.425348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.065086 (+1639738us) service_pool.cc:227] Handling call
0927 23:31:19.070655 (+ 5569us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.075006 (+ 4351us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.075151 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.088766 1656 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:19.067948 3761 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4.
I20260927 23:31:19.094456 1627 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:19.103713 3809 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.012s sys 0.013s
W20260927 23:31:19.104164 3809 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.918s user 0.012s sys 0.013s
W20260927 23:31:19.120864 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 48672.4us
I20260927 23:31:19.122119 1585 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 23:31:19.128371 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1374) took 2022 ms. Trace:
I20260927 23:31:19.128918 1585 rpcz_store.cc:276] 0927 23:31:17.105532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.062853 (+1957321us) service_pool.cc:227] Handling call
0927 23:31:19.073027 (+ 10174us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:19.128047 (+ 55020us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.128302 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.129683 1627 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:19.082870 3470 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:19.091547 1702 connection.cc:232] RPC call timeout handler was delayed by -1.1260792500000321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
W20260927 23:31:19.132031 1702 connection.cc:232] RPC call timeout handler was delayed by -1.050179331000038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
I20260927 23:31:19.133595 1663 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:19.135433 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:19.131250 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 61933.7us
I20260927 23:31:19.090624 3134 raft_consensus.cc:686] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:19.148524 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:19.149441 2259 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:19.119688 3642 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.153788 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.148032 1663 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:19.130946 3696 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.192678 2259 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:19.155277 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:19.197307 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:19.198851 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 43666us
W20260927 23:31:19.201642 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57459.2us
I20260927 23:31:19.206584 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:19.197933 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.148461 3776 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:19.193917 3665 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:19.122157 3770 consensus_queue.cc:1035] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.080s
I20260927 23:31:19.215121 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.201953 3763 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:19.205385 3842 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:19.197340 3470 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:19.236299 1702 leader_election.cc:334] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.928s (SENT)
I20260927 23:31:19.244125 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:19.213124 1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1644) took 4595 ms (client timeout 1577 ms). Trace:
I20260927 23:31:19.237836 1732 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:19.246557 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.152717 3591 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
W20260927 23:31:19.244858 1661 rpcz_store.cc:269] 0927 23:31:14.600560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.725024 (+124464us) service_pool.cc:227] Handling call
0927 23:31:19.150431 (+4425407us) spinlock_profiling.cc:243] Waited 2.55 s on lock 0x7b54003f3754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:19.195506 (+ 45075us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5859390720}
I20260927 23:31:19.213991 3757 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260927 23:31:19.242362 3839 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.152s user 0.000s sys 0.001s
I20260927 23:31:19.213255 1654 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:19.287542 1732 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:19.215230 3616 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:19.227658 3741 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:19.279299 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.289652 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.301033 1733 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:19.286069 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:19.290023 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 189789us
I20260927 23:31:19.285604 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:19.241336 3763 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:19.211867 3642 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 23:31:19.284950 2289 pending_rounds.cc:76] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting all ops after (but not including) 5
I20260927 23:31:19.243898 3842 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:19.309662 3841 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:19.310418 3824 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:19.311249 3831 log.cc:924] Time spent T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.291s user 0.001s sys 0.001s
I20260927 23:31:19.310999 3730 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.159s
I20260927 23:31:19.282963 3757 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:19.314582 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.314831 3747 raft_consensus.cc:2781] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:19.314913 1551 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:19.281548 3470 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.284564 3134 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.316164 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.411440 1734 raft_consensus.cc:2396] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:19.316360 1627 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:19.316701 3809 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:19.316881 3758 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.317348 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.318926 3822 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.324138 3439 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:19.324476 1628 leader_election.cc:334] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.627s (SENT)
I20260927 23:31:19.325088 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.326042 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.326566 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.330585 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:19.206028 1657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08bafa5142a9452db744d03134c85be6"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
mode: GRACEFUL
new_leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
from {username='slave'} at 127.0.0.1:37356
I20260927 23:31:19.334918 3830 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.338613 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:19.434664 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.435889 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:19.436187 3843 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:19.437063 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:19.340567 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 125368us
W20260927 23:31:19.440474 1702 leader_election.cc:334] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.187s (SENT)
I20260927 23:31:19.440865 1702 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:19.341230 3832 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20260927 23:31:19.308091 1586 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:19.400039 3781 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:19.310091 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46178 (request call id 891) took 3906 ms. Trace:
I20260927 23:31:19.473757 1659 rpcz_store.cc:276] 0927 23:31:15.384795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.246135 (+3861340us) service_pool.cc:227] Handling call
0927 23:31:19.290984 (+ 44849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.321983 3492 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.353724 1656 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:19.291448 3811 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:19.357090 1588 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:19.487015 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.286922 3713 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:19.286674 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.400602 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.490439 2291 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:19.343003 3763 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:19.356525 3837 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
W20260927 23:31:19.415395 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 165387us
W20260927 23:31:19.418340 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 103506us
I20260927 23:31:19.510074 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.514350 1551 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:19.515543 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75126.5us
I20260927 23:31:19.515717 3616 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:19.414659 1585 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:19.363634 3824 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:19.406656 3747 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:19.425140 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 108847us
I20260927 23:31:19.364252 3842 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:19.401769 3757 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:19.526599 1580 raft_consensus.cc:2354] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:19.356057 2289 pending_rounds.cc:97] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260927 23:31:19.427963 3822 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:19.428426 3439 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.431769 3840 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:19.432219 1628 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:19.432746 1733 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:19.434801 1736 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:19.339738 1730 raft_consensus.cc:1261] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:19.396572 3464 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.471112 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.337504 1582 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:19.521531 3824 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:19.524118 3747 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:19.410043 3841 raft_consensus.cc:686] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:19.442634 3809 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.482800 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:19.413997 3825 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:19.493333 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 204823us
I20260927 23:31:19.424332 3755 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:19.485908 3758 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:19.356854 3665 raft_consensus.cc:686] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:19.504383 3848 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
W20260927 23:31:19.498226 3666 consensus_meta.cc:287] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.672s user 0.008s sys 0.004s
I20260927 23:31:19.511211 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.591805 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.536077 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.471808 1657 raft_consensus.cc:595] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Received request to transfer leadership to TS 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:19.593152 1657 raft_consensus.cc:603] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:19.567544 1581 raft_consensus.cc:1261] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:19.541962 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.533341 3705 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:19.595350 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 270897us
I20260927 23:31:19.538097 3811 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:19.579689 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.600524 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37356 (request call id 6) took 3095 ms. Trace:
I20260927 23:31:19.600893 1657 rpcz_store.cc:276] 0927 23:31:16.503944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.193350 (+2689406us) service_pool.cc:227] Handling call
0927 23:31:19.593527 (+400177us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:19.580056 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1302) took 2674 ms (client timeout 2238 ms). Trace:
W20260927 23:31:19.590036 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 107401us
W20260927 23:31:19.602294 1587 rpcz_store.cc:269] 0927 23:31:16.893080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.258281 (+1365201us) service_pool.cc:227] Handling call
0927 23:31:19.566985 (+1308704us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b5400061154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:19.567096 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":887443840}
I20260927 23:31:19.602954 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:19.529996 2289 raft_consensus.cc:2866] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:19.603610 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.604487 1585 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:19.573482 3758 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:19.607743 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.601809 3666 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.572615 3464 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.095s
I20260927 23:31:19.572636 3825 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.595718 3134 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:19.743537 1702 leader_election.cc:334] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.899s (SENT)
W20260927 23:31:19.672690 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 57718.4us
I20260927 23:31:19.573467 3591 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:19.674037 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.759572 1580 raft_consensus.cc:2362] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4. Re-sending same reply.
I20260927 23:31:19.752647 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1315) took 1268 ms. Trace:
I20260927 23:31:19.573537 3755 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:19.692188 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 89318.6us
I20260927 23:31:19.674809 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:19.724357 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 9c719671f35640549c1b944112b3fc03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260927 23:31:19.762490 1585 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:19.065857 1583 rpcz_store.cc:276] 0927 23:31:17.087531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.004687 (+1917156us) service_pool.cc:227] Handling call
0927 23:31:19.004938 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.060347 (+ 55409us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:19.060385 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:19.060425 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:19.060465 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.060531 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.060665 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23260,"raft.queue_time_us":54046,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":3768}
I20260927 23:31:19.726039 3809 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.601672 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.729017 3665 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.724301 2292 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260927 23:31:19.732035 1552 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:56820 dropped due to backpressure. The service queue is full; it has 50 items.
I20260927 23:31:19.763728 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.764519 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.765309 2292 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260927 23:31:19.732795 3492 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:19.685369 3864 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20260927 23:31:19.686265 3865 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260927 23:31:19.734419 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1662) took 2614 ms. Trace:
I20260927 23:31:19.733295 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.736102 3666 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:19.684221 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1259) took 3125 ms. Trace:
I20260927 23:31:19.738754 2290 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:19.769578 3439 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.683387 3781 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:19.767943 3755 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:19.788890 1584 rpcz_store.cc:276] 0927 23:31:16.374633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:16.374754 (+ 121us) service_pool.cc:227] Handling call
0927 23:31:16.375166 (+ 412us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:16.375559 (+ 393us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:16.375608 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:16.393946 (+ 18338us) log.cc:842] Serialized 52 byte log entry
0927 23:31:18.336494 (+1942548us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b18003b6080. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db29345 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f6411 00007f9750a435cf
0927 23:31:18.506408 (+169914us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:18.506472 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.602787 (+ 96315us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:19.482019 (+879232us) raft_consensus.cc:1671] finished
0927 23:31:19.482042 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.500213 (+ 18171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8837,"raft.queue_time_us":71424,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":22414,"spinlock_wait_cycles":3144960,"thread_start_us":98802,"threads_started":1,"wal-append.queue_time_us":2546366}
I20260927 23:31:19.791600 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.792733 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.793136 3842 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.793776 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.739372 3464 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:19.794572 3464 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.788398 3843 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:19.796357 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.796808 3464 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:19.643599 3758 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:19.739934 3867 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260927 23:31:19.610419 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.799244 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.799562 2285 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:19.573395 3840 raft_consensus.cc:686] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:19.801452 3840 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.802021 3755 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.802985 3470 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:19.788618 3861 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260927 23:31:19.804816 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1314) took 1402 ms. Trace:
I20260927 23:31:19.582048 3696 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:19.805239 1737 rpcz_store.cc:276] 0927 23:31:18.416612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.499926 (+ 83314us) service_pool.cc:227] Handling call
0927 23:31:18.603078 (+103152us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:18.603718 (+ 640us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:18.948815 (+345097us) log.cc:842] Serialized 14 byte log entry
0927 23:31:19.608628 (+659813us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:19.687885 (+ 79257us) log.cc:842] Serialized 26 byte log entry
0927 23:31:19.688034 (+ 149us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:19.688071 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.688143 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:19.804463 (+116320us) raft_consensus.cc:1671] finished
0927 23:31:19.804487 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.804633 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":423198,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":100854,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"spinlock_wait_cycles":64067968,"thread_start_us":970776,"threads_started":3,"wal-append.queue_time_us":544637}
I20260927 23:31:19.704216 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.807425 3825 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:19.644613 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.808465 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.738957 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.808806 1588 raft_consensus.cc:2362] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3. Re-sending same reply.
I20260927 23:31:19.609870 1579 raft_consensus.cc:2354] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
I20260927 23:31:19.763645 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:19.769146 1470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 78238.3us
I20260927 23:31:19.770392 1554 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:19.796995 1580 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:19.806588 3696 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.760341 1729 rpcz_store.cc:276] 0927 23:31:18.484405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.484710 (+ 305us) service_pool.cc:227] Handling call
0927 23:31:18.492935 (+ 8225us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:18.493183 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:18.493219 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:18.493715 (+ 496us) log.cc:842] Serialized 26 byte log entry
0927 23:31:18.949823 (+456108us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:18.949870 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:18.949959 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:19.644760 (+694801us) raft_consensus.cc:1671] finished
0927 23:31:19.644788 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.752578 (+107790us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":452240,"threads_started":1,"wal-append.queue_time_us":492428}
I20260927 23:31:19.764213 1552 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:56820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1421)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1429)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1416)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1409)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1410)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1419)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1400)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1417)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1426)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1387)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 18)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 306)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1380)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1386)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1411)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1414)
I20260927 23:31:19.603860 3639 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:19.809343 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 214) took 1371 ms. Trace:
I20260927 23:31:19.812726 1663 rpcz_store.cc:276] 0927 23:31:18.437889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.154975 (+717086us) service_pool.cc:227] Handling call
0927 23:31:19.738791 (+583816us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:19.809298 (+ 70507us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85631744}
I20260927 23:31:19.814005 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.815290 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.789321 1553 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:56820 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1396)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1395)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1428)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1415)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1437)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1400)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1431)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1432)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 312)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1427)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1422)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1418)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1424)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1423)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 309)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1438)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1417)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1436)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1435)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1389)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1430)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1426)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1434)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1388)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1394)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1439)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1433)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1387)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 18)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 306)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1380)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1385)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1386)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1404)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1411)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1414)
I20260927 23:31:19.812158 1587 raft_consensus.cc:2354] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
I20260927 23:31:19.817355 2259 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:19.817302 3639 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.819195 3639 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:19.819281 3862 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:19.812703 1658 rpcz_store.cc:276] 0927 23:31:17.119639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.485170 (+1365531us) service_pool.cc:227] Handling call
0927 23:31:18.785501 (+300331us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:18.905040 (+119539us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.734177 (+829137us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54002b07d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:19.734359 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":268033792}
I20260927 23:31:19.812219 2294 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:19.828923 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.829818 1588 raft_consensus.cc:2354] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5.
I20260927 23:31:19.830607 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.831605 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.716454 3872 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:19.835785 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:19.839082 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 95599.2us
I20260927 23:31:19.837540 1579 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:19.842054 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:19.602423 3705 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:19.842140 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.843143 1658 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:19.819677 3862 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:19.846525 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.845505 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.823113 3616 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:19.848053 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.844544 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1664) took 2724 ms. Trace:
I20260927 23:31:19.848469 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.851950 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.852416 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.853605 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.854133 1579 raft_consensus.cc:2354] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
I20260927 23:31:19.853842 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:19.854660 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 251113us
W20260927 23:31:19.855552 1702 leader_election.cc:334] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.183s (SENT)
I20260927 23:31:19.856081 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:19.849627 3862 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:19.873842 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.865852 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.851603 3616 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:19.867115 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.885614 2259 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:19.833504 3858 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260927 23:31:19.906131 3868 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:19.906824 3134 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:19.913120 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.913786 1657 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:19.907255 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.908064 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1501) took 1902 ms. Trace:
I20260927 23:31:19.910485 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1316) took 2808 ms. Trace:
I20260927 23:31:19.915313 1581 rpcz_store.cc:276] 0927 23:31:17.102545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.061411 (+1958866us) service_pool.cc:227] Handling call
0927 23:31:19.567197 (+505786us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.593797 (+ 26600us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.910276 (+316479us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b5400061150. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:19.910423 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1162334464}
I20260927 23:31:19.917750 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.687719 3841 consensus_queue.cc:237] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:19.931598 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 85361.4us
W20260927 23:31:19.913457 2257 leader_election.cc:334] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.505s (SENT)
I20260927 23:31:19.932463 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:19.838344 3747 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.916915 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:19.920933 2290 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 476) took 1515 ms (client timeout 1865 ms). Trace:
I20260927 23:31:19.922484 3757 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:19.923445 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:19.919236 1659 rpcz_store.cc:276] 0927 23:31:18.006020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.474948 (+1468928us) service_pool.cc:227] Handling call
0927 23:31:19.503099 (+ 28151us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.511969 (+ 8870us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:19.760130 (+248161us) log.cc:842] Serialized 14 byte log entry
0927 23:31:19.907716 (+147586us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:19.907775 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:19.907819 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.907900 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.908013 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":245011,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":90,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":281672,"threads_started":2}
I20260927 23:31:19.925602 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.873404 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.927192 3687 raft_consensus.cc:3032] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:19.929544 3475 raft_consensus.cc:3032] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:19.930359 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.958987 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 218) took 1470 ms. Trace:
I20260927 23:31:19.876718 3616 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:19.940124 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.991139 1585 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:19.993419 1581 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:19.990901 1657 rpcz_store.cc:276] 0927 23:31:18.489075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.957337 (+1468262us) service_pool.cc:227] Handling call
0927 23:31:19.958505 (+ 1168us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.958704 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:19.958734 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:19.958775 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:19.958801 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.958850 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.958941 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.943477 1588 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:19.994925 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:19.995653 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.997315 2291 raft_consensus.cc:1226] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 23:31:19.958616 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:19.963563 1702 leader_election.cc:334] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.062s (SENT)
I20260927 23:31:19.999722 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:20.003189 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:20.006675 1655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 216) took 1515 ms (client timeout 1842 ms). Trace:
W20260927 23:31:20.007094 1655 rpcz_store.cc:269] 0927 23:31:18.488419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.897175 (+408756us) service_pool.cc:227] Handling call
0927 23:31:20.003596 (+1106421us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b54000dc0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.003693 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64721792}
I20260927 23:31:19.933712 3825 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:20.009765 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.848956 3872 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.007630 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:20.012403 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.012599 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:19.954862 3878 log.cc:924] Time spent T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260927 23:31:20.054260 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.054487 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.040015 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.054370 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.056728 1585 raft_consensus.cc:2362] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4. Re-sending same reply.
I20260927 23:31:20.058180 1588 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:19.992743 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:19.953243 3689 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:19.931005 3822 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:20.036154 1736 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:20.056818 1659 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 4
I20260927 23:31:20.013197 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:20.089524 1655 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:20.056110 1580 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
W20260927 23:31:20.015864 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 73675.8us
I20260927 23:31:20.107041 1629 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.079595 1737 raft_consensus.cc:1261] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:20.083303 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.992695 2285 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:20.110759 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.998484 3687 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:19.991673 3475 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:20.191177 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 220) took 1701 ms. Trace:
I20260927 23:31:20.191640 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:20.192085 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 278652us
I20260927 23:31:20.194535 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1668) took 1633 ms. Trace:
W20260927 23:31:20.107550 1700 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.510s (SENT) [suppressed 55 similar messages]
I20260927 23:31:20.195037 1655 rpcz_store.cc:276] 0927 23:31:18.561694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.008042 (+1446348us) service_pool.cc:227] Handling call
0927 23:31:20.086225 (+ 78183us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.194218 (+107993us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.194468 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:20.195937 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 88428.4us
W20260927 23:31:20.196624 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 89707.3us
I20260927 23:31:20.191689 1658 rpcz_store.cc:276] 0927 23:31:18.489636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.006944 (+1517308us) service_pool.cc:227] Handling call
0927 23:31:20.060085 (+ 53141us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.085652 (+ 25567us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.085692 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:20.190813 (+105121us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:20.190882 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.190968 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.191110 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.109959 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:19.912031 3763 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:20.112100 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 99640.8us
I20260927 23:31:19.958688 3755 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:20.197613 1629 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:19.939049 3134 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:19.962625 3638 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.153s
I20260927 23:31:20.213165 1729 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:20.217876 1728 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:20.009529 3842 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:20.083987 1654 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:20.222151 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.222412 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.199182 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1379) took 1093 ms. Trace:
I20260927 23:31:19.960397 3824 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.190213 3503 raft_consensus.cc:3032] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:20.086975 3689 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:20.248970 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 51476us
I20260927 23:31:20.230904 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.250226 1629 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.250707 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1388) took 1142 ms. Trace:
I20260927 23:31:20.251135 1583 rpcz_store.cc:276] 0927 23:31:19.108516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.193042 (+1084526us) service_pool.cc:227] Handling call
0927 23:31:20.250625 (+ 57583us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:19.939375 2290 rpcz_store.cc:269] 0927 23:31:18.405092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.405386 (+ 294us) service_pool.cc:227] Handling call
0927 23:31:19.920705 (+1515319us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54006419d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:19.920853 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3063209216}
W20260927 23:31:20.232254 3876 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260927 23:31:20.234652 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:20.261840 1655 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:20.238610 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.059057 3811 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:20.223078 3642 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:20.190661 2288 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:20.243003 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:20.266441 1663 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:20.266860 1658 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:20.246071 1579 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:20.245870 1585 rpcz_store.cc:276] 0927 23:31:19.105296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.058359 (+953063us) service_pool.cc:227] Handling call
0927 23:31:20.199117 (+140758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.273784 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1389) took 1164 ms. Trace:
I20260927 23:31:20.224011 3696 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.286223 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.287389 1583 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:20.305789 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.306665 1580 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:20.274351 1581 rpcz_store.cc:276] 0927 23:31:19.108859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.109198 (+1000339us) service_pool.cc:227] Handling call
0927 23:31:20.273697 (+164499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.239442 3872 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.319875 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:20.320533 1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 221) took 1830 ms (client timeout 1505 ms). Trace:
I20260927 23:31:20.320767 3475 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.250270 3439 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:20.255362 3492 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.104s
I20260927 23:31:20.325848 1659 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:20.330379 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.261394 3875 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:20.332736 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.252233 3689 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:20.190544 3822 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.248600 3503 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:20.223122 3825 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:20.362174 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1284) took 3494 ms (client timeout 1805 ms). Trace:
I20260927 23:31:20.499163 1658 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:20.363376 1656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 215) took 1925 ms (client timeout 2008 ms). Trace:
I20260927 23:31:20.348259 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:20.523649 1656 rpcz_store.cc:269] 0927 23:31:18.438241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.937853 (+499612us) service_pool.cc:227] Handling call
0927 23:31:20.363202 (+1425349us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54004466d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.363319 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291773824}
I20260927 23:31:20.520534 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.524739 1580 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
W20260927 23:31:20.525614 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 257398us
I20260927 23:31:20.526432 1629 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.467557 1736 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:20.502925 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.475162 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.321720 3811 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:20.522043 1662 rpcz_store.cc:269] 0927 23:31:18.489900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.769858 (+279958us) service_pool.cc:227] Handling call
0927 23:31:20.320319 (+1550461us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b54001cfcd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.320464 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":169012992}
I20260927 23:31:20.476867 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.273958 3705 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:20.540195 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:20.548749 1588 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:20.274758 3664 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:20.547477 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:20.563632 1656 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:20.563437 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.565210 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.551270 1729 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
W20260927 23:31:20.505326 3879 log.cc:924] Time spent T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.280s user 0.000s sys 0.000s
W20260927 23:31:20.522856 1582 rpcz_store.cc:269] 0927 23:31:16.867733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:17.182221 (+314488us) service_pool.cc:227] Handling call
0927 23:31:20.361940 (+3179719us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b5400465d54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.362085 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4891875072}
W20260927 23:31:20.358541 3758 consensus_meta.cc:287] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.001s
I20260927 23:31:20.552464 1658 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:20.566962 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.567085 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:20.474092 3707 consensus_meta.cc:287] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.380s user 0.012s sys 0.000s
I20260927 23:31:20.568315 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:20.568789 1551 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:20.574855 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.573597 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.576848 1729 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:20.577011 1579 raft_consensus.cc:2396] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:20.570133 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:20.570495 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 606932us
I20260927 23:31:20.356637 3824 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:20.578143 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1387) took 1470 ms. Trace:
I20260927 23:31:20.577122 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:20.590564 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1669) took 1949 ms. Trace:
I20260927 23:31:20.590909 1658 rpcz_store.cc:276] 0927 23:31:18.623283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.551871 (+1928588us) service_pool.cc:227] Handling call
0927 23:31:20.552250 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.572237 (+ 19987us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.572467 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.591311 1582 rpcz_store.cc:276] 0927 23:31:19.108174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.569933 (+1461759us) service_pool.cc:227] Handling call
0927 23:31:20.578096 (+ 8163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.367556 3763 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.579663 3777 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:20.580523 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1671) took 1684 ms. Trace:
I20260927 23:31:20.581415 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.592101 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1394) took 1454 ms. Trace:
I20260927 23:31:20.252328 3862 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.578441 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.594378 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:20.620253 1702 leader_election.cc:334] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.656s (SENT)
I20260927 23:31:20.620725 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:20.579319 3707 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.579560 3665 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260927 23:31:20.595611 3591 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:20.596724 3825 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:20.595618 1659 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:20.630945 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1673) took 1724 ms. Trace:
I20260927 23:31:20.631459 1659 rpcz_store.cc:276] 0927 23:31:18.906851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.571978 (+1665127us) service_pool.cc:227] Handling call
0927 23:31:20.595265 (+ 23287us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.630576 (+ 35311us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.630881 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.592209 3761 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:20.590849 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1397) took 1309 ms. Trace:
I20260927 23:31:20.633018 1588 rpcz_store.cc:276] 0927 23:31:19.281102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.558786 (+1277684us) service_pool.cc:227] Handling call
0927 23:31:20.590795 (+ 32009us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.579907 1585 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:20.566223 3638 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:20.605894 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 227) took 1363 ms. Trace:
I20260927 23:31:20.605472 1656 rpcz_store.cc:276] 0927 23:31:18.887641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.563126 (+1675485us) service_pool.cc:227] Handling call
0927 23:31:20.563426 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.572178 (+ 8752us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.572412 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:19.850654 1661 rpcz_store.cc:276] 0927 23:31:17.120308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.437948 (+2317640us) service_pool.cc:227] Handling call
0927 23:31:19.438247 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:19.472106 (+ 33859us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:19.603629 (+131523us) log.cc:842] Serialized 14 byte log entry
0927 23:31:19.844138 (+240509us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:19.844202 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:19.844259 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.844340 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:19.844477 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":209663,"prepare.run_cpu_time_us":2633,"prepare.run_wall_time_us":11960,"raft.queue_time_us":24391,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":4871,"thread_start_us":245448,"threads_started":2}
I20260927 23:31:20.607146 1580 rpcz_store.cc:276] 0927 23:31:19.138154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.525506 (+1387352us) service_pool.cc:227] Handling call
0927 23:31:20.592035 (+ 66529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.613060 3811 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.618979 3654 consensus_queue.cc:1046] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:20.619364 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1507) took 1013 ms. Trace:
I20260927 23:31:20.622445 1729 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:20.568102 2289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 479) took 1409 ms. Trace:
I20260927 23:31:20.626307 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.577642 3754 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:20.576747 3757 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.589347 3475 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:20.629056 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.631915 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:20.634399 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.518121 3616 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.577437 3758 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.673635 1585 raft_consensus.cc:1226] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 2. Current term is 4. Ops: []
I20260927 23:31:20.674116 1588 raft_consensus.cc:1261] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:20.673306 1663 rpcz_store.cc:276] 0927 23:31:19.605930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.815001 (+209071us) service_pool.cc:227] Handling call
0927 23:31:20.619201 (+804200us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54002b11d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.619314 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1035374464}
I20260927 23:31:20.415246 3687 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.625983 3117 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:20.679574 2258 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.698187 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:20.738631 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 59164.2us
I20260927 23:31:20.697890 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:20.697487 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:20.740770 1581 raft_consensus.cc:2396] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260927 23:31:20.742955 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 122739us
I20260927 23:31:20.743053 1581 raft_consensus.cc:1261] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:20.721201 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 502628us
I20260927 23:31:20.735771 1472 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0be2a211b7754af89c38527623e5d967 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 23:31:20.745200 1579 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:20.721791 1730 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:20.648749 1735 raft_consensus.cc:2379] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:20.685904 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:20.744531 1658 raft_consensus.cc:2379] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:20.740924 1661 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:20.708945 1583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0be2a211b7754af89c38527623e5d967"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:20.666513 1703 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:20.747857 1583 raft_consensus.cc:595] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.676134 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:20.685141 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:20.751572 1579 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:20.753201 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.754580 3663 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:20.768148 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 18) took 1036 ms. Trace:
I20260927 23:31:20.768762 1584 raft_consensus.cc:2354] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:20.768857 1583 rpcz_store.cc:276] 0927 23:31:19.728158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.630045 (+901887us) service_pool.cc:227] Handling call
0927 23:31:20.764375 (+134330us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.770787 1662 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260927 23:31:20.771481 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:20.772420 1580 raft_consensus.cc:1226] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 1. Current term is 3. Ops: []
I20260927 23:31:20.639515 3761 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:20.777395 1700 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:20.780031 1628 connection.cc:232] RPC call timeout handler was delayed by -1.7450578610000207s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260927 23:31:20.780866 1628 leader_election.cc:334] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.545s (SENT)
I20260927 23:31:20.761749 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 306) took 1843 ms. Trace:
I20260927 23:31:20.763352 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1380) took 1657 ms. Trace:
I20260927 23:31:20.783439 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 225) took 1643 ms. Trace:
I20260927 23:31:20.783176 1585 rpcz_store.cc:276] 0927 23:31:19.105640 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.634039 (+1528399us) service_pool.cc:227] Handling call
0927 23:31:20.649529 (+ 15490us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.763038 (+113509us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.763282 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.784250 1658 rpcz_store.cc:276] 0927 23:31:19.140105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.747493 (+1607388us) service_pool.cc:227] Handling call
0927 23:31:20.782766 (+ 35273us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.783008 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.783049 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:20.783109 (+ 60us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:20.783156 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.783226 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.783370 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.783524 1588 rpcz_store.cc:276] 0927 23:31:18.901440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.633598 (+1732158us) service_pool.cc:227] Handling call
0927 23:31:20.650077 (+ 16479us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.744365 (+ 94288us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.744593 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.766414 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1404) took 1452 ms. Trace:
I20260927 23:31:20.786510 1579 rpcz_store.cc:276] 0927 23:31:19.313534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.750946 (+1437412us) service_pool.cc:227] Handling call
0927 23:31:20.751338 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.764977 (+ 13639us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.766343 (+ 1366us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.788375 1579 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260927 23:31:20.748790 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 82358us
I20260927 23:31:20.648671 3665 consensus_queue.cc:1225] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260927 23:31:20.637916 3638 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:20.770053 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1385) took 1636 ms. Trace:
W20260927 23:31:20.794154 3732 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.049s sys 0.024s
I20260927 23:31:20.800625 1581 rpcz_store.cc:276] 0927 23:31:19.107456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.741807 (+1634351us) service_pool.cc:227] Handling call
0927 23:31:20.742664 (+ 857us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.744006 (+ 1342us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.744267 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.764523 2292 raft_consensus.cc:2379] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:20.803123 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1386) took 1695 ms. Trace:
I20260927 23:31:20.803871 1580 rpcz_store.cc:276] 0927 23:31:19.107829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.745393 (+1637564us) service_pool.cc:227] Handling call
0927 23:31:20.772207 (+ 26814us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.802797 (+ 30590us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.803049 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.760444 3811 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:20.742142 3850 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:20.651108 3873 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:20.797314 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1411) took 1371 ms. Trace:
I20260927 23:31:20.837386 1579 rpcz_store.cc:276] 0927 23:31:19.423124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.787533 (+1364409us) service_pool.cc:227] Handling call
0927 23:31:20.787940 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.788815 (+ 875us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.789094 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.834297 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 313) took 1016 ms. Trace:
I20260927 23:31:20.839077 1579 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:20.839835 1579 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:20.758080 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:20.807273 3777 consensus_queue.cc:1035] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260927 23:31:20.842738 3873 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.809681 3638 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:20.676172 2289 rpcz_store.cc:276] 0927 23:31:19.141489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.144575 (+ 3086us) service_pool.cc:227] Handling call
0927 23:31:19.234214 (+ 89639us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:19.686691 (+452477us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:19.686735 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:19.687306 (+ 571us) log.cc:842] Serialized 52 byte log entry
0927 23:31:19.859571 (+172265us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:19.864996 (+ 5425us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.865098 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:20.520279 (+655181us) raft_consensus.cc:1671] finished
0927 23:31:20.520302 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.550954 (+ 30652us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2391,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7988,"thread_start_us":162382,"threads_started":1,"wal-append.queue_time_us":526467}
I20260927 23:31:20.810205 3811 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:20.849922 3811 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:20.851104 3884 log.cc:924] Time spent T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
W20260927 23:31:20.852922 3883 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260927 23:31:20.809478 3665 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260927 23:31:20.638095 1654 rpcz_store.cc:276] 0927 23:31:19.242595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.325777 (+ 83182us) service_pool.cc:227] Handling call
0927 23:31:20.599954 (+1274177us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b5400352354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:20.600097 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1289586432}
I20260927 23:31:20.854923 1654 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:20.855630 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1682) took 1337 ms. Trace:
I20260927 23:31:20.856031 1654 rpcz_store.cc:276] 0927 23:31:19.518578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.854269 (+1335691us) service_pool.cc:227] Handling call
0927 23:31:20.854687 (+ 418us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.855343 (+ 656us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.855567 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.857609 1654 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:20.838825 1585 rpcz_store.cc:276] 0927 23:31:19.817512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.798617 (+981105us) service_pool.cc:227] Handling call
0927 23:31:20.833644 (+ 35027us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.833868 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.833907 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:20.833963 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:20.834008 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.834077 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.834222 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.861719 3811 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.861490 1579 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.864111 3811 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:20.864701 1579 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:20.866415 1660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.033s sys 0.048s
W20260927 23:31:20.866762 1660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.033s sys 0.048s
I20260927 23:31:20.867333 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1683) took 1339 ms. Trace:
I20260927 23:31:20.869516 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 316) took 1021 ms. Trace:
I20260927 23:31:20.781340 1628 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:20.814385 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1681) took 1614 ms. Trace:
I20260927 23:31:20.870431 1658 rpcz_store.cc:276] 0927 23:31:19.199999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.785672 (+1585673us) service_pool.cc:227] Handling call
0927 23:31:20.813726 (+ 28054us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.813952 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.813991 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:20.814047 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:20.814088 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.814165 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.814300 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.836625 1580 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:20.634980 3591 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.874217 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:20.874506 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 315) took 1050 ms. Trace:
I20260927 23:31:20.875042 1580 rpcz_store.cc:276] 0927 23:31:19.824005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.805053 (+981048us) service_pool.cc:227] Handling call
0927 23:31:20.836393 (+ 31340us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.874175 (+ 37782us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.874437 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.789925 3687 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:20.813279 3745 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:20.771508 3654 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:20.782097 3776 consensus_queue.cc:1046] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260927 23:31:20.792692 3663 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.651239 3825 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:20.801210 3732 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.049s sys 0.024s
W20260927 23:31:20.826946 1702 leader_election.cc:334] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.745s (SENT)
I20260927 23:31:20.870118 1583 rpcz_store.cc:276] 0927 23:31:19.848572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.805333 (+956761us) service_pool.cc:227] Handling call
0927 23:31:20.868893 (+ 63560us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.869111 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.869149 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:20.869206 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:20.869256 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.869327 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:20.869451 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.812538 1588 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:20.905043 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1414) took 1399 ms. Trace:
I20260927 23:31:20.905469 1588 rpcz_store.cc:276] 0927 23:31:19.506006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.790708 (+1284702us) service_pool.cc:227] Handling call
0927 23:31:20.812178 (+ 21470us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.904702 (+ 92524us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.904979 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.869357 1654 rpcz_store.cc:276] 0927 23:31:19.518903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.857039 (+1338136us) service_pool.cc:227] Handling call
0927 23:31:20.857394 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.858034 (+ 640us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.858269 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:20.797821 3840 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:20.751680 3117 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.645394 3503 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:20.901664 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 120843us
I20260927 23:31:20.927109 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.883013 3492 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:20.934607 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.936442 1730 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
W20260927 23:31:20.937423 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 184004us
W20260927 23:31:20.943303 1628 leader_election.cc:334] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.805s (SENT)
W20260927 23:31:20.946199 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 119280us
I20260927 23:31:20.946599 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:20.946771 3503 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:20.947218 1730 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:20.936028 3761 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:20.929000 3840 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:20.873631 3850 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260927 23:31:20.945176 3862 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:20.902334 3654 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:20.960446 3840 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:21.008519 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 223497us
I20260927 23:31:20.960155 3707 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:21.040786 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1312) took 2639 ms. Trace:
I20260927 23:31:21.041088 1658 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:21.047859 1579 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:21.054303 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 165938us
I20260927 23:31:21.043291 1731 rpcz_store.cc:276] 0927 23:31:18.401821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.401942 (+ 121us) service_pool.cc:227] Handling call
0927 23:31:18.402288 (+ 346us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:18.420414 (+ 18126us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:18.420443 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:18.483933 (+ 63490us) log.cc:842] Serialized 52 byte log entry
0927 23:31:18.981415 (+497482us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:20.599223 (+1617808us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b18003a3cc0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db44155 00007f9754444167 00007f9750a79807 00007f9750a88f0f
0927 23:31:20.599348 (+ 125us) log.cc:842] Serialized 14 byte log entry
0927 23:31:20.994110 (+394762us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.994188 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.002111 (+ 7923us) raft_consensus.cc:1671] finished
0927 23:31:21.024178 (+ 22067us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.040691 (+ 16513us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12603,"prepare.queue_time_us":105242,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":17020,"raft.queue_time_us":19392,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":73443,"spinlock_wait_cycles":14680320,"thread_start_us":735433,"threads_started":3,"wal-append.queue_time_us":535121,"wal-append.run_cpu_time_us":3162,"wal-append.run_wall_time_us":1298272}
W20260927 23:31:21.057463 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 265542us
I20260927 23:31:21.058849 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.062649 1728 raft_consensus.cc:2396] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260927 23:31:21.060863 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 254017us
W20260927 23:31:21.059746 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 123358us
W20260927 23:31:21.073756 1628 connection.cc:232] RPC call timeout handler was delayed by -1.6572540020000588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1867s for a response
W20260927 23:31:21.055948 3883 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260927 23:31:21.042735 3616 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:21.078747 1628 leader_election.cc:334] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.995s (SENT)
I20260927 23:31:21.043684 3758 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.083403 1628 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:21.076234 3693 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.010s sys 0.080s
I20260927 23:31:21.089083 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.091975 1731 raft_consensus.cc:3037] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.093215 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.079389 3841 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:21.095624 1733 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 23:31:21.093955 1700 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.086988 3862 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260927 23:31:21.094245 1728 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:21.314138 1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1398) took 2002 ms (client timeout 1775 ms). Trace:
W20260927 23:31:21.363215 1587 rpcz_store.cc:269] 0927 23:31:19.311726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.603320 (+291594us) service_pool.cc:227] Handling call
0927 23:31:21.313948 (+1710628us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54003093d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:21.314073 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477520768}
I20260927 23:31:21.323544 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1684) took 1804 ms. Trace:
I20260927 23:31:21.373059 1658 rpcz_store.cc:276] 0927 23:31:19.519221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.871646 (+1352425us) service_pool.cc:227] Handling call
0927 23:31:20.894687 (+ 23041us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.055165 (+160478us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.323289 (+268124us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002f2bd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:21.323480 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":278966528}
I20260927 23:31:21.374259 1736 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
W20260927 23:31:21.091483 3693 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.742s user 0.010s sys 0.081s
W20260927 23:31:21.368371 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 425317us
I20260927 23:31:21.337635 1729 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:21.131435 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:21.314036 1730 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260927 23:31:21.252647 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 173905us
W20260927 23:31:21.363651 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 269932us
W20260927 23:31:21.253249 3898 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20260927 23:31:21.304441 3689 consensus_meta.cc:287] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.731s user 0.010s sys 0.000s
I20260927 23:31:21.119038 3525 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:21.381513 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:21.256309 3888 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
I20260927 23:31:21.301129 3907 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:21.378767 1554 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:21.389823 1733 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:21.300231 3901 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:21.428324 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 49666.2us
I20260927 23:31:21.249415 3663 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.411046 2294 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:21.384307 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1323) took 2145 ms. Trace:
W20260927 23:31:21.382222 3475 consensus_meta.cc:287] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.610s user 0.011s sys 0.006s
I20260927 23:31:21.094898 3758 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:21.378779 3654 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:21.098750 3745 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:21.420305 1658 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 23:31:21.208333 3591 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.409938 1700 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.370587 3825 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:21.384016 3894 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20260927 23:31:21.429996 1735 rpcz_store.cc:276] 0927 23:31:19.238808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.239030 (+ 222us) service_pool.cc:227] Handling call
0927 23:31:21.384136 (+2145106us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400148054. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:21.384254 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3077276928}
W20260927 23:31:21.412412 1628 connection.cc:232] RPC call timeout handler was delayed by -1.6487578610000355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23090000000000002s for a response
I20260927 23:31:21.431941 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:21.476311 2292 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:21.432431 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1485) took 5967 ms. Trace:
I20260927 23:31:21.432948 1736 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:21.433661 1729 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:21.434702 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:21.433699 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:21.435863 1733 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:21.436257 3908 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:21.437477 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:21.479920 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 46292.3us
I20260927 23:31:21.430796 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 314) took 1607 ms. Trace:
I20260927 23:31:21.472886 3745 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:21.481031 1628 leader_election.cc:334] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.447s (SENT)
W20260927 23:31:21.481709 1628 connection.cc:232] RPC call timeout handler was delayed by -1.5762540020000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20260927 23:31:21.482036 1628 connection.cc:232] RPC call timeout handler was delayed by -1.5482746040000279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28940000000000005s for a response
I20260927 23:31:21.481073 1581 rpcz_store.cc:276] 0927 23:31:19.823676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.801860 (+978184us) service_pool.cc:227] Handling call
0927 23:31:20.813393 (+ 11533us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.813648 (+ 255us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:21.030508 (+216860us) log.cc:842] Serialized 14 byte log entry
0927 23:31:21.425720 (+395212us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:21.425790 (+ 70us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:21.425853 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.425937 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.430733 (+ 4796us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1233,"prepare.queue_time_us":272797,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":37,"raft.queue_time_us":204721,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":577512,"threads_started":3,"wal-append.queue_time_us":338043}
I20260927 23:31:21.445761 2294 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:21.410629 1730 pending_rounds.cc:76] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 2
I20260927 23:31:21.259696 3616 consensus_queue.cc:1046] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.166s
I20260927 23:31:21.383325 3723 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:21.468920 1731 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
W20260927 23:31:21.470345 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 60659.5us
I20260927 23:31:21.469031 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1685) took 1657 ms. Trace:
I20260927 23:31:21.496220 1658 rpcz_store.cc:276] 0927 23:31:19.811955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.374481 (+1562526us) service_pool.cc:227] Handling call
0927 23:31:21.419908 (+ 45427us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.468744 (+ 48836us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.468975 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:21.498008 1658 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:21.482303 1628 connection.cc:232] RPC call timeout handler was delayed by -1.5168540020000592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
W20260927 23:31:21.499177 1628 connection.cc:232] RPC call timeout handler was delayed by -1.5141540020000548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
I20260927 23:31:21.499348 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1513) took 1654 ms. Trace:
I20260927 23:31:21.499853 1658 rpcz_store.cc:276] 0927 23:31:19.845221 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.497200 (+1651979us) service_pool.cc:227] Handling call
0927 23:31:21.497587 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.499063 (+ 1476us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.499268 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:21.500250 1628 connection.cc:232] RPC call timeout handler was delayed by -1.0191540020000502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2576s for a response
I20260927 23:31:21.444478 3665 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.473737 3707 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:21.483989 1733 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:21.482842 1736 raft_consensus.cc:1203] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20260927 23:31:21.484724 1703 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.505478 1736 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 23:31:21.224248 3904 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.244s
I20260927 23:31:21.483393 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:21.485790 1729 raft_consensus.cc:1203] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20260927 23:31:21.511117 1729 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260927 23:31:21.486505 1661 consensus_meta.cc:287] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.725s user 0.007s sys 0.011s
I20260927 23:31:21.511579 1661 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:21.512363 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.483078 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.513285 1735 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:21.467789 3822 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:21.384181 3841 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.477146 1660 rpcz_store.cc:276] 0927 23:31:15.465103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.590839 (+3125736us) service_pool.cc:227] Handling call
0927 23:31:20.306135 (+1715296us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:20.306392 (+ 257us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.331931 (+ 25539us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.331976 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:20.332418 (+ 442us) log.cc:842] Serialized 26 byte log entry
0927 23:31:20.867209 (+534791us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:20.867277 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.867356 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.432170 (+564814us) raft_consensus.cc:1671] finished
0927 23:31:21.432187 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.432377 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19323,"raft.queue_time_us":41806,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":15126400,"thread_start_us":534403,"threads_started":1,"wal-append.queue_time_us":661217}
I20260927 23:31:21.483805 1730 pending_rounds.cc:97] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 23:31:21.503490 1581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:21.518349 1730 raft_consensus.cc:2866] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:21.518615 1581 raft_consensus.cc:595] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:21.429090 3689 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.519590 1730 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 23:31:21.514439 2257 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:21.520965 3689 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.514746 3654 raft_consensus.cc:3032] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:21.521561 3654 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:21.510032 3665 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.513842 1728 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:21.490545 3591 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:21.522276 1628 leader_election.cc:334] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.817s (SENT)
I20260927 23:31:21.517023 1656 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:21.528730 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:21.510617 2291 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:21.512887 1661 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:21.482386 3745 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:21.509282 3908 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.531342 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:21.493161 3503 consensus_meta.cc:287] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.546s user 0.010s sys 0.001s
I20260927 23:31:21.522665 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1504) took 1921 ms. Trace:
I20260927 23:31:21.533026 1733 raft_consensus.cc:1261] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:21.545863 1586 raft_consensus.cc:2354] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4.
W20260927 23:31:21.546543 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1413) took 2062 ms (client timeout 1603 ms). Trace:
W20260927 23:31:21.546959 1586 rpcz_store.cc:269] 0927 23:31:19.483886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.605332 (+121446us) service_pool.cc:227] Handling call
0927 23:31:21.531284 (+1925952us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b5c00179c8c. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97543b6361 00007f97545501f9 00007f97559b0ac6 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:21.546302 (+ 15018us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37298560}
I20260927 23:31:21.444185 3907 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.553449 2294 raft_consensus.cc:1226] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 23:31:21.554107 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1680) took 2412 ms. Trace:
I20260927 23:31:21.555534 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.554625 1659 rpcz_store.cc:276] 0927 23:31:19.141635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.752395 (+1610760us) service_pool.cc:227] Handling call
0927 23:31:20.793158 (+ 40763us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:20.793421 (+ 263us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:20.837495 (+ 44074us) log.cc:842] Serialized 14 byte log entry
0927 23:31:21.482491 (+644996us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b18001eb260. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db29345 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f657b 00007f9750a7988a
0927 23:31:21.553689 (+ 71198us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:21.553755 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:21.553817 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.553899 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.554043 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":43872,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"spinlock_wait_cycles":77806336,"thread_start_us":70965,"threads_started":1}
I20260927 23:31:21.543856 1654 rpcz_store.cc:276] 0927 23:31:19.601650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.910365 (+1308715us) service_pool.cc:227] Handling call
0927 23:31:20.937457 (+ 27092us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.945903 (+ 8446us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.945937 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:20.947677 (+ 1740us) log.cc:842] Serialized 26 byte log entry
0927 23:31:21.336901 (+389224us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:21.381713 (+ 44812us) log.cc:842] Serialized 14 byte log entry
0927 23:31:21.383917 (+ 2204us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.383997 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.522477 (+138480us) raft_consensus.cc:1671] finished
0927 23:31:21.522497 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.522613 (+ 116us) inbound_call.cc:173] Queueing success response
0927 23:31:21.525581 (+ 2968us) spinlock_profiling.cc:243] Waited 35.2 ms on lock 0x7b1800035fe0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
Metrics: {"mutex_wait_us":8038,"prepare.queue_time_us":44412,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":10528,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":12079,"spinlock_wait_cycles":152923264,"thread_start_us":388740,"threads_started":1,"wal-append.queue_time_us":428834}
I20260927 23:31:21.563041 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.563159 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.541075 2289 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:21.523968 3525 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:21.568744 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 46497.2us
W20260927 23:31:21.569584 1628 leader_election.cc:334] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.867s (SENT)
I20260927 23:31:21.471727 3901 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.527181 3763 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:21.581506 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.548373 3850 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.551591 3758 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:21.556239 3745 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:21.525238 3665 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.504038 3723 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.590219 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:21.584780 3906 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.325s user 0.000s sys 0.001s
I20260927 23:31:21.599166 1654 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:21.598615 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:21.603297 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 118669us
I20260927 23:31:21.595566 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:21.604110 1661 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:21.541460 3475 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260927 23:31:21.600767 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.601639 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.602991 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:21.605580 1728 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.604156 2258 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.609232 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:21.609966 1659 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.611443 1703 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.584213 3763 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:21.619830 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 50288.9us
W20260927 23:31:21.621362 1628 leader_election.cc:334] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.974s (SENT)
I20260927 23:31:21.605861 1586 raft_consensus.cc:595] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:21.522423 3654 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.630965 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.595184 3525 raft_consensus.cc:686] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:21.637612 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.637755 3758 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.638669 3922 consensus_queue.cc:1035] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:21.641185 3763 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.660537 3763 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:21.660768 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1676) took 2731 ms. Trace:
I20260927 23:31:21.661106 1663 rpcz_store.cc:276] 0927 23:31:18.929822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.736915 (+1807093us) service_pool.cc:227] Handling call
0927 23:31:20.765910 (+ 28995us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.831665 (+ 65755us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.831707 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:20.832286 (+ 579us) log.cc:842] Serialized 26 byte log entry
0927 23:31:21.059085 (+226799us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:21.059141 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.059229 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.660565 (+601336us) raft_consensus.cc:1671] finished
0927 23:31:21.660587 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.660713 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35247,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":39188,"thread_start_us":123523,"threads_started":1,"wal-append.queue_time_us":212274}
I20260927 23:31:21.662681 3643 raft_consensus.cc:3032] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:21.663060 3643 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:21.663867 3643 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.665010 3643 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:21.668148 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1678) took 2723 ms. Trace:
I20260927 23:31:21.668488 1656 rpcz_store.cc:276] 0927 23:31:18.945312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.738776 (+1793464us) service_pool.cc:227] Handling call
0927 23:31:21.513637 (+774861us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.530530 (+ 16893us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.667956 (+137426us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54002f30d0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:21.668092 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1719067008}
I20260927 23:31:21.669898 1656 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260927 23:31:21.671653 3891 log.cc:924] Time spent T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.669s user 0.002s sys 0.000s
W20260927 23:31:21.673334 3886 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.632s user 0.003s sys 0.000s
I20260927 23:31:21.674217 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 291) took 1459 ms. Trace:
I20260927 23:31:21.674585 1737 rpcz_store.cc:276] 0927 23:31:20.214900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.215639 (+ 739us) service_pool.cc:227] Handling call
0927 23:31:20.215996 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.244805 (+ 28809us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.244846 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:20.252865 (+ 8019us) log.cc:842] Serialized 26 byte log entry
0927 23:31:20.649208 (+396343us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:20.649246 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.649318 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.671022 (+1021704us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:21.673888 (+ 2866us) raft_consensus.cc:1671] finished
0927 23:31:21.673914 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.674144 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24834,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":167362432,"thread_start_us":231876,"threads_started":1,"wal-append.queue_time_us":654824}
I20260927 23:31:21.676297 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.677026 1737 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:21.678875 3914 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260927 23:31:21.605381 3707 raft_consensus.cc:686] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:21.681732 3890 log.cc:924] Time spent T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.694s user 0.003s sys 0.000s
I20260927 23:31:21.608194 3745 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:21.586308 3464 consensus_queue.cc:1046] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260927 23:31:21.690400 3745 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.691588 3757 raft_consensus.cc:3032] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:21.691954 3757 raft_consensus.cc:729] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:21.628582 1627 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:21.692798 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 64317.1us
I20260927 23:31:21.693567 3875 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:21.693992 3875 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:21.694306 3875 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:21.694512 3693 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260927 23:31:21.698006 3840 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.644s user 0.011s sys 0.005s
I20260927 23:31:21.698835 3654 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:21.703855 3901 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:21.705327 3723 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:21.594347 3850 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.708197 3850 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.620954 3822 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:21.709108 3822 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.711911 3822 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:21.647329 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.712414 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:21.712970 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 91646.6us
I20260927 23:31:21.714758 1485 catalog_manager.cc:5495] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:21.717315 3907 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:21.718935 3841 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:21.719106 1628 leader_election.cc:334] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.576s (SENT)
W20260927 23:31:21.719322 1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1400) took 2407 ms (client timeout 2183 ms). Trace:
W20260927 23:31:21.719717 1584 rpcz_store.cc:269] 0927 23:31:19.312382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.845865 (+533483us) service_pool.cc:227] Handling call
0927 23:31:21.719138 (+1873273us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b5400466c54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:21.719260 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2094761344}
I20260927 23:31:21.624445 3900 raft_consensus.cc:2781] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:21.721335 2286 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:21.721976 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 417) took 1117 ms. Trace:
I20260927 23:31:21.719691 1628 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:21.722574 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:21.723129 3841 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:21.724150 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1672) took 2832 ms. Trace:
I20260927 23:31:21.724589 3862 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.724474 1662 rpcz_store.cc:276] 0927 23:31:18.892163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.567569 (+1675406us) service_pool.cc:227] Handling call
0927 23:31:20.593035 (+ 25466us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.794530 (+201495us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.723934 (+929404us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540013a1d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:21.724092 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":407988608}
I20260927 23:31:21.723153 1586 raft_consensus.cc:595] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:21.648495 1661 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:21.655993 3921 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260927 23:31:21.658105 3913 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260927 23:31:21.658485 3758 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.659595 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.685074 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1516) took 1717 ms. Trace:
W20260927 23:31:21.694012 3917 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260927 23:31:21.686898 1730 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:31:21.747390 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:21.643152 1702 leader_election.cc:334] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.712s (SENT)
I20260927 23:31:21.688066 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 219) took 3198 ms. Trace:
I20260927 23:31:21.749701 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:21.688326 3916 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260927 23:31:21.689368 3638 consensus_meta.cc:287] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.841s user 0.003s sys 0.007s
I20260927 23:31:21.689070 3755 consensus_queue.cc:1046] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:21.548419 3908 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.759444 3908 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.694257 1627 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:21.702047 1484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 149) took 11660 ms (client timeout 15000 ms). Trace:
W20260927 23:31:21.760341 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 66176.8us
I20260927 23:31:21.761678 3841 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:21.760234 1484 rpcz_store.cc:269] 0927 23:31:10.041835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:10.043398 (+ 1563us) service_pool.cc:227] Handling call
0927 23:31:21.701831 (+11658433us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:15.441916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:15.442591 (+ 675us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:15.442651 (+ 60us) write_op.cc:435] Acquired schema lock
0927 23:31:15.442665 (+ 14us) tablet.cc:660] Decoding operations
0927 23:31:15.443822 (+ 1157us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:15.443913 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:15.443934 (+ 21us) tablet.cc:683] Acquiring locks for 11 operations
0927 23:31:15.445395 (+ 1461us) tablet.cc:699] Row locks acquired
0927 23:31:15.445429 (+ 34us) write_op.cc:260] PREPARE: finished
0927 23:31:15.445596 (+ 167us) write_op.cc:270] Start()
0927 23:31:15.445796 (+ 200us) write_op.cc:276] Timestamp: P: 1790551875445537 usec, L: 0
0927 23:31:15.445815 (+ 19us) op_driver.cc:331] REPLICATION: starting
0927 23:31:15.524777 (+ 78962us) log.cc:842] Serialized 4268 byte log entry
0927 23:31:16.488831 (+964054us) op_driver.cc:444] REPLICATION: finished
0927 23:31:16.822170 (+333339us) write_op.cc:301] APPLY: starting
0927 23:31:16.822271 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:16.822637 (+ 366us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:16.822801 (+ 164us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:16.824330 (+ 1529us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:16.824624 (+ 294us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0927 23:31:16.824668 (+ 44us) write_op.cc:312] APPLY: finished
0927 23:31:16.825002 (+ 334us) log.cc:842] Serialized 83 byte log entry
0927 23:31:16.825137 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:16.825238 (+ 101us) write_op.cc:454] Released schema lock
0927 23:31:16.825270 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:16.825474 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:16.825494 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":298,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":322793,"apply.run_cpu_time_us":3606,"apply.run_wall_time_us":3607,"num_ops":11,"prepare.queue_time_us":842354,"prepare.run_cpu_time_us":73986,"prepare.run_wall_time_us":314637,"raft.queue_time_us":293223,"raft.run_cpu_time_us":16207,"raft.run_wall_time_us":353090,"replication_time_us":1020361,"thread_start_us":1074589,"threads_started":4,"wal-append.queue_time_us":378948,"wal-append.run_cpu_time_us":33516,"wal-append.run_wall_time_us":2172913}]]}
W20260927 23:31:21.708285 3887 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.668s user 0.001s sys 0.001s
I20260927 23:31:21.727953 1728 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
W20260927 23:31:21.771770 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 9c719671f35640549c1b944112b3fc03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260927 23:31:21.746933 3901 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.730221 1735 raft_consensus.cc:2396] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:21.748315 1657 rpcz_store.cc:276] 0927 23:31:18.489355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.994227 (+1504872us) service_pool.cc:227] Handling call
0927 23:31:20.055031 (+ 60804us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:20.055401 (+ 370us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:20.055431 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:20.117864 (+ 62433us) log.cc:842] Serialized 26 byte log entry
0927 23:31:20.633047 (+515183us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:20.633084 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:20.633145 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.491865 (+858720us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:21.682891 (+191026us) raft_consensus.cc:1671] finished
0927 23:31:21.682918 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.687991 (+ 5073us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81835,"raft.queue_time_us":755,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":25849,"spinlock_wait_cycles":194301312,"thread_start_us":372040,"threads_started":1,"wal-append.queue_time_us":657880}
I20260927 23:31:21.776719 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.777297 1735 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.778295 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:31:21.762467 3841 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:21.740460 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.779345 3841 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.780035 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.780953 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:21.781162 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.781884 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:21.744153 1656 rpcz_store.cc:276] 0927 23:31:19.968091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.669341 (+1701250us) service_pool.cc:227] Handling call
0927 23:31:21.669695 (+ 354us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:21.670220 (+ 525us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.685021 (+ 14801us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:21.742262 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:21.744167 1731 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:21.722218 3591 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:21.753952 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.734298 3921 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:21.772243 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.717675 1485 catalog_manager.cc:5495] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:21.786720 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.793074 1728 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
W20260927 23:31:21.773655 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 130530us
I20260927 23:31:21.734309 3900 raft_consensus.cc:686] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:21.782011 1661 raft_consensus.cc:2396] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:21.686108 1659 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:21.797801 1663 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 23:31:21.794483 1735 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:21.742780 3525 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.799891 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.796903 1654 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:21.788502 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 58841.3us
I20260927 23:31:21.801965 1735 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:21.725139 3862 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.771522 3866 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:21.803797 2258 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:21.803921 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:21.801573 1730 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:21.739943 3840 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:21.797896 3754 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.112s user 0.015s sys 0.003s
I20260927 23:31:21.811367 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.812674 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.819437 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.820035 1731 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:21.820221 1735 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:21.813999 1661 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:21.815315 1628 leader_election.cc:334] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.894s (SENT)
W20260927 23:31:21.815991 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 62616.9us
I20260927 23:31:21.823632 1551 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:21.817375 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.825574 1730 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:21.798153 3907 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.860626 1628 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:21.860637 1485 catalog_manager.cc:5495] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 3, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:21.800678 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:21.797479 3921 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.741108 3707 consensus_queue.cc:237] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:21.862665 1485 catalog_manager.cc:5495] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 2, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 2 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:21.771950 3638 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.865733 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:21.898384 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.903469 1728 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:21.729560 2286 rpcz_store.cc:276] 0927 23:31:20.604101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.605967 (+ 1866us) service_pool.cc:227] Handling call
0927 23:31:21.720933 (+1114966us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:21.721119 (+ 186us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.721723 (+ 604us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.721919 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29620096}
I20260927 23:31:21.899101 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.906487 1731 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:21.868373 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.900748 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.908061 1735 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:21.790964 3901 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.820722 3777 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:21.819839 3503 raft_consensus.cc:3032] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:21.916651 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 101379us
W20260927 23:31:21.917316 1628 leader_election.cc:334] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.208s (SENT)
W20260927 23:31:21.924863 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 59219.1us
I20260927 23:31:21.932780 1655 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260927 23:31:21.931229 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:21.899895 3776 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:21.933624 1737 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:21.792613 3757 consensus_queue.cc:260] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.901401 3761 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:21.898911 2287 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:21.862352 3475 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260927 23:31:21.951030 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:21.958041 1735 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:21.921336 1485 catalog_manager.cc:5495] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 3, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:21.905975 3665 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.919623 3777 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:21.920127 3503 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:21.870647 3907 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:21.947113 2285 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:21.912410 2290 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 418) took 1124 ms. Trace:
I20260927 23:31:21.968394 1587 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:21.969660 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.028549 1586 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:21.970525 3777 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:21.987141 1581 raft_consensus.cc:3032] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:22.038834 1581 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:22.038872 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:21.998436 1580 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:21.999857 3757 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:22.003050 1735 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:21.991763 3849 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:21.998446 1731 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:21.984929 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.042590 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:22.013931 1728 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:22.045853 1581 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.051931 3907 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:22.052682 1485 catalog_manager.cc:5495] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 2, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 2 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:22.010882 3761 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.054407 3535 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:22.056320 3862 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:22.056895 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64820.1us
I20260927 23:31:22.044358 1586 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:21.968083 3665 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.109717 1581 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:22.111032 1588 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:22.112377 3840 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.991724 3897 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.123s
I20260927 23:31:22.042215 3758 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:21.953804 3900 consensus_queue.cc:237] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.163532 1735 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:22.165736 1485 catalog_manager.cc:5495] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:22.121883 3755 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.187466 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1515) took 2309 ms. Trace:
I20260927 23:31:22.173894 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:22.192533 1735 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:22.127331 3747 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.191152 1658 rpcz_store.cc:276] 0927 23:31:19.877698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.500739 (+1623041us) service_pool.cc:227] Handling call
0927 23:31:21.501097 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.552890 (+ 51793us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:21.581871 (+ 28981us) log.cc:842] Serialized 14 byte log entry
0927 23:31:21.813681 (+231810us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:21.813742 (+ 61us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:21.813800 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.187139 (+373339us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b54003af7d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:22.187201 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:22.187393 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33656,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":38,"raft.queue_time_us":807,"spinlock_wait_cycles":285195776,"thread_start_us":230522,"threads_started":1,"wal-append.queue_time_us":354131}
I20260927 23:31:22.211750 1485 catalog_manager.cc:5495] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:22.214380 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.178743 3934 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
I20260927 23:31:22.016810 2290 rpcz_store.cc:276] 0927 23:31:20.788276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.794449 (+ 6173us) service_pool.cc:227] Handling call
0927 23:31:20.873861 (+ 79412us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:21.135678 (+261817us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:21.137808 (+ 2130us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:21.141006 (+ 3198us) log.cc:842] Serialized 26 byte log entry
0927 23:31:21.430969 (+289963us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:21.445517 (+ 14548us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.468309 (+ 22792us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:21.892541 (+424232us) raft_consensus.cc:1671] finished
0927 23:31:21.896888 (+ 4347us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:21.909158 (+ 12270us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":31009,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":27948,"thread_start_us":429407,"threads_started":2,"wal-append.queue_time_us":361143}
I20260927 23:31:22.168704 3946 consensus_queue.cc:1035] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260927 23:31:22.216786 3117 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:22.217200 3940 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260927 23:31:22.109355 3503 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.232962 3758 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:22.233484 3503 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:22.234097 3758 raft_consensus.cc:3032] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:22.234483 3758 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:22.235255 3758 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.177407 3873 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.219786 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.236310 3758 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:22.228834 1586 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:22.227056 3755 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.240979 3755 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.242094 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:22.178743 3931 log.cc:924] Time spent T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260927 23:31:22.175184 3930 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260927 23:31:22.174548 3927 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260927 23:31:22.229844 1731 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260927 23:31:22.165359 3654 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.116847 3849 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.231577 3643 raft_consensus.cc:979] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:22.231950 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:22.125451 3761 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.255026 1581 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:22.256455 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:22.257452 3915 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.661s user 0.000s sys 0.001s
I20260927 23:31:22.291581 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:22.257314 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.258646 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:22.259479 1485 catalog_manager.cc:5495] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:22.293550 3862 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:22.294104 1485 catalog_manager.cc:5495] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 4, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:22.295473 1485 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 3, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:22.267611 3665 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.274451 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.298849 3643 raft_consensus.cc:1067] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:22.299844 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.300436 1587 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:22.301214 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.301728 1587 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.276475 1580 raft_consensus.cc:1226] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 3. Ops: []
W20260927 23:31:22.226486 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 108337us
I20260927 23:31:22.167124 3776 consensus_queue.cc:1046] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260927 23:31:22.311823 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.253616 3875 consensus_meta.cc:287] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.559s user 0.006s sys 0.003s
I20260927 23:31:22.253950 1730 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.313237 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.319129 1580 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:22.320477 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.273695 1731 pending_rounds.cc:76] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 2
I20260927 23:31:22.289305 2259 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.288367 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:22.322508 1731 pending_rounds.cc:97] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 23:31:22.323104 1735 raft_consensus.cc:3037] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:22.290290 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.290548 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.290915 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.306945 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.310736 1702 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.311718 1657 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:22.313397 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.309253 1586 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:22.326402 2285 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 415) took 1714 ms (client timeout 1598 ms). Trace:
I20260927 23:31:22.327018 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.332086 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 302) took 1003 ms. Trace:
I20260927 23:31:22.338269 3761 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.351509 1586 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:22.358995 3880 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 2.120s user 0.007s sys 0.008s
I20260927 23:31:22.333384 3875 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.340549 1580 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:22.338975 3904 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:22.333467 1588 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:22.336190 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 47013.9us
I20260927 23:31:22.368716 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1332) took 3052 ms. Trace:
I20260927 23:31:22.342715 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:22.374958 2259 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:22.347200 1587 raft_consensus.cc:1261] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 23:31:22.323151 1731 raft_consensus.cc:2866] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:22.337226 2289 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:22.374966 1730 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:22.382671 1658 raft_consensus.cc:2396] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 23:31:22.373515 1734 rpcz_store.cc:276] 0927 23:31:19.315941 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.415833 (+ 99892us) service_pool.cc:227] Handling call
0927 23:31:19.474817 (+ 58984us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:19.623752 (+148935us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:19.623801 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:19.698684 (+ 74883us) log.cc:842] Serialized 52 byte log entry
0927 23:31:19.930279 (+231595us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:19.930311 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:19.930401 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:22.316717 (+2386316us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:22.368418 (+ 51701us) raft_consensus.cc:1671] finished
0927 23:31:22.368439 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:22.368648 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":50078,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":32924,"spinlock_wait_cycles":9173760,"thread_start_us":234028,"threads_started":2,"wal-append.queue_time_us":313641}
I20260927 23:31:22.371127 3875 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.338127 3431 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:22.391745 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:22.384714 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.381196 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 91037.6us
W20260927 23:31:22.390260 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 76167.3us
W20260927 23:31:22.393046 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 82366.9us
I20260927 23:31:22.394093 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.395980 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.356642 3687 consensus_meta.cc:287] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.464s user 0.017s sys 0.004s
I20260927 23:31:22.370432 1732 rpcz_store.cc:276] 0927 23:31:21.328730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.328895 (+ 165us) service_pool.cc:227] Handling call
0927 23:31:21.329569 (+ 674us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:21.329890 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:21.329917 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:21.355577 (+ 25660us) log.cc:842] Serialized 26 byte log entry
0927 23:31:21.471499 (+115922us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:21.471565 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:21.471656 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:22.256891 (+785235us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:22.305307 (+ 48416us) raft_consensus.cc:1671] finished
0927 23:31:22.305328 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:22.332019 (+ 26691us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":476,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":13198,"spinlock_wait_cycles":19185408,"thread_start_us":112343,"threads_started":1,"wal-append.queue_time_us":159052}
I20260927 23:31:22.394272 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:22.246487 3638 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.392179 3431 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:22.402381 1702 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.382006 3901 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.403049 3822 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:22.340055 3825 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:22.412101 1582 raft_consensus.cc:2396] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:22.282084 3117 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.104s
I20260927 23:31:22.426496 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.365960 3921 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:22.427779 3895 log.cc:924] Time spent T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 1.013s user 0.005s sys 0.000s
I20260927 23:31:22.285917 3654 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.386024 3747 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260927 23:31:22.404841 3757 consensus_queue.cc:1046] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260927 23:31:22.430389 3875 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:22.433058 1728 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:22.392598 3873 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:22.416457 1587 raft_consensus.cc:1261] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:22.417251 1732 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:22.417346 3947 consensus_queue.cc:1035] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:31:22.398525 3840 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:22.375628 2285 rpcz_store.cc:269] 0927 23:31:20.601935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.606263 (+ 4328us) service_pool.cc:227] Handling call
0927 23:31:20.757860 (+151597us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:22.305452 (+1547592us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540029ed54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:22.315956 (+ 10504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2428783232}
I20260927 23:31:22.422190 3862 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:22.428931 1581 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.432916 3757 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:22.415418 3925 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:22.425024 1735 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:22.431532 1627 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.418956 3777 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.441829 1587 raft_consensus.cc:1261] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:22.442444 2258 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.434027 3825 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:22.444346 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1597) took 1010 ms. Trace:
I20260927 23:31:22.446058 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.423919 3763 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.446792 1588 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:22.446926 3763 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.449491 1737 raft_consensus.cc:1261] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 23:31:22.451383 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 49103.7us
I20260927 23:31:22.452795 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1687) took 1851 ms. Trace:
I20260927 23:31:22.453145 1663 rpcz_store.cc:276] 0927 23:31:20.600950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.684643 (+1083693us) service_pool.cc:227] Handling call
0927 23:31:21.797588 (+112945us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:21.800379 (+ 2791us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.452634 (+652255us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54002b07d0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:22.452746 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242699136}
W20260927 23:31:22.453588 1702 leader_election.cc:334] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.227s (SENT)
I20260927 23:31:22.455003 1657 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:22.457005 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1598) took 1022 ms. Trace:
I20260927 23:31:22.444756 1733 rpcz_store.cc:276] 0927 23:31:21.434042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.521576 (+ 87534us) service_pool.cc:227] Handling call
0927 23:31:21.532637 (+ 11061us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.545245 (+ 12608us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.444183 (+898938us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b54002bd750. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:22.444297 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24772352}
I20260927 23:31:22.454005 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.460569 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.467703 1736 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:22.479564 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.480520 1735 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 23:31:22.483386 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.484534 3676 raft_consensus.cc:2781] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:22.485026 3676 raft_consensus.cc:686] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:22.457343 1729 rpcz_store.cc:276] 0927 23:31:21.442697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.518880 (+ 76183us) service_pool.cc:227] Handling call
0927 23:31:21.519115 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.606550 (+ 87435us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:21.644360 (+ 37810us) log.cc:842] Serialized 14 byte log entry
0927 23:31:21.857579 (+213219us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:21.857647 (+ 68us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:21.857718 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.456801 (+599083us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b540011c454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:22.456842 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:22.456953 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":30089,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":121,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":29625,"spinlock_wait_cycles":185303936,"thread_start_us":212627,"threads_started":1,"wal-append.queue_time_us":312778}
I20260927 23:31:22.492844 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.496978 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:22.470424 3642 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.002s sys 0.024s
I20260927 23:31:22.513340 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.444738 3840 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:22.503835 1583 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:22.522583 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.523370 1583 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:22.510963 1627 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.519575 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.518304 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.527415 1587 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.466274 3873 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.492058 3591 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:22.491560 3696 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260927 23:31:22.533929 1628 leader_election.cc:334] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.854s (SENT)
I20260927 23:31:22.535701 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.536743 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.549883 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.550675 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.551843 1586 raft_consensus.cc:2396] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:22.501225 3707 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:22.512709 3642 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.003s sys 0.024s
I20260927 23:31:22.514051 3638 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:22.520555 3758 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:22.532735 3840 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:22.485895 3676 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.518925 3776 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.547076 1627 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:22.566731 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 46813us
I20260927 23:31:22.533349 3683 consensus_queue.cc:1035] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260927 23:31:22.567843 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.530473 3946 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:22.593061 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.533162 3875 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:22.546168 3693 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:22.591359 1583 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:22.581672 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.624780 1587 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:22.558651 3654 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.560266 3591 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:22.564683 3638 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:22.593545 1586 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:22.629498 1627 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:22.629760 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.653584 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.630235 1585 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:22.639801 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:22.625379 3906 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260927 23:31:22.641324 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.593396 3758 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:22.540278 3873 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.623091 3947 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:22.644718 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:22.628193 1702 leader_election.cc:334] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.510s (SENT)
W20260927 23:31:22.661937 1655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1519) took 2404 ms (client timeout 2713 ms). Trace:
W20260927 23:31:22.703032 1655 rpcz_store.cc:269] 0927 23:31:20.249259 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.263168 (+ 13909us) service_pool.cc:227] Handling call
0927 23:31:22.653841 (+2390673us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000bd6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:22.653971 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3158712576}
I20260927 23:31:22.703846 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.704908 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.699905 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.706301 1586 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:22.708449 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:22.709810 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 81665.1us
W20260927 23:31:22.710795 1702 leader_election.cc:334] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.620s (SENT)
I20260927 23:31:22.711632 1654 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260927 23:31:22.711659 1702 leader_election.cc:334] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.746s (SENT)
I20260927 23:31:22.713284 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.712634 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.715987 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.659310 3910 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:22.656131 3638 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:22.716827 1585 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:22.719177 1702 leader_election.cc:334] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.441s (SENT)
I20260927 23:31:22.710183 3921 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.715266 3822 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:22.708999 3891 log.cc:924] Time spent T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260927 23:31:22.732604 1587 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.642233 3875 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:22.739364 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 224782us
I20260927 23:31:22.772266 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.627643 3776 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.773466 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.785487 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.797513 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:22.797820 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 78680.5us
I20260927 23:31:22.714465 3757 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:22.720117 3849 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.753156 3901 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:22.733335 3910 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:22.846822 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.829909 1551 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.847646 1583 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:22.783638 3777 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:22.832973 1700 leader_election.cc:334] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.551s (SENT)
I20260927 23:31:22.833778 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.855012 1700 leader_election.cc:334] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.620s (SENT)
I20260927 23:31:22.849823 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.855517 1700 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.631296 3946 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:22.856776 1700 leader_election.cc:334] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.671s (SENT)
I20260927 23:31:22.855531 1586 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:22.858709 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.859925 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:22.857492 1700 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 1.926s (SENT)
I20260927 23:31:22.853099 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.861889 1583 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:22.851758 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.863006 1587 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:22.863739 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.864603 1586 raft_consensus.cc:2362] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:22.865757 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:22.867054 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.871989 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:22.873759 1657 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:22.872774 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.867753 1661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:31:22.878953 1661 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:22.879311 1661 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:22.823809 3757 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:22.882588 3904 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:22.883041 3951 log.cc:924] Time spent T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
W20260927 23:31:22.883747 3949 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20260927 23:31:22.887563 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:22.892308 3642 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:22.873761 3761 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:22.894614 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 96142us
I20260927 23:31:22.896432 3776 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:22.896620 3946 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.899695 3873 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:22.902694 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 251574us
I20260927 23:31:22.906054 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.906888 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:22.921821 1657 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:22.829487 3503 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.084s
I20260927 23:31:22.903888 1700 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:22.926982 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.927474 1654 raft_consensus.cc:1261] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:22.927788 1585 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.910938 1584 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:22.929454 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.932094 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.910821 1661 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.914961 1586 raft_consensus.cc:2379] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:22.873859 3901 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:22.908527 1583 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:22.908257 3757 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:22.924120 1628 leader_election.cc:334] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.792s (SENT)
W20260927 23:31:22.925282 3953 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
I20260927 23:31:22.925812 3921 consensus_queue.cc:1035] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:22.910178 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.947386 1587 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:22.937305 1628 leader_election.cc:334] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.836s (SENT)
I20260927 23:31:22.937485 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.935824 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.948201 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:22.936401 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.934661 1584 raft_consensus.cc:2379] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:22.944160 1661 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:22.853847 3758 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:22.957108 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.830263 3591 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.971728 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.983448 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1358) took 1031 ms. Trace:
I20260927 23:31:22.983326 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:22.984099 1584 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:22.959722 1583 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260927 23:31:22.919155 3642 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:22.985037 1733 rpcz_store.cc:276] 0927 23:31:21.952039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.478530 (+526491us) service_pool.cc:227] Handling call
0927 23:31:22.478834 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.479114 (+ 280us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:22.677719 (+198605us) log.cc:842] Serialized 14 byte log entry
0927 23:31:22.983140 (+305421us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:22.983188 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:22.983230 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.983284 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:22.983397 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31009,"prepare.queue_time_us":389462,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":39,"thread_start_us":386933,"threads_started":2}
I20260927 23:31:23.003050 1585 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 5. Ops: [3.4-3.4]
I20260927 23:31:22.957621 3901 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:23.032256 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 324) took 1064 ms. Trace:
I20260927 23:31:23.035017 1579 rpcz_store.cc:276] 0927 23:31:21.940315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.940433 (+ 118us) service_pool.cc:227] Handling call
0927 23:31:21.940796 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.030057 (+ 89261us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:22.030387 (+ 330us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:22.034042 (+ 3655us) log.cc:842] Serialized 26 byte log entry
0927 23:31:22.475667 (+441625us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:22.475704 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.475765 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:23.004254 (+528489us) raft_consensus.cc:1671] finished
0927 23:31:23.004281 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.004449 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68629,"raft.queue_time_us":74101,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6348,"thread_start_us":346941,"threads_started":1,"wal-append.queue_time_us":467263}
I20260927 23:31:23.033974 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46178 (request call id 892) took 7642 ms. Trace:
I20260927 23:31:23.037058 1660 rpcz_store.cc:276] 0927 23:31:15.387367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.715825 (+6328458us) service_pool.cc:227] Handling call
0927 23:31:22.945540 (+1229715us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002f3ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4af0 00007f9750a77f2a 00007f97559b4c6e 00007f974f0ab807 00007f974f0ab792 00007f974f0ab6c1 00007f974f0ab63c 00007f974f0aa8f2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:23.029903 (+ 84363us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1531333504}
I20260927 23:31:23.038226 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.038993 1733 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:23.002960 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65701.8us
I20260927 23:31:22.933941 3873 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260927 23:31:22.914099 3904 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:23.042281 1628 leader_election.cc:334] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.192s (SENT)
I20260927 23:31:23.063966 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1353) took 1254 ms. Trace:
I20260927 23:31:22.948055 3707 consensus_queue.cc:1035] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20260927 23:31:22.986351 3776 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:22.934998 3638 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.080874 3841 raft_consensus.cc:3032] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:23.081385 3841 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:22.986577 3825 consensus_meta.cc:287] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
I20260927 23:31:23.082587 3825 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.084962 3825 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:22.933774 3925 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260927 23:31:22.984021 3693 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:23.071780 1732 rpcz_store.cc:276] 0927 23:31:21.809949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.438515 (+628566us) service_pool.cc:227] Handling call
0927 23:31:22.468043 (+ 29528us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.468294 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:22.721267 (+252973us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.063492 (+342225us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:23.063564 (+ 72us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:23.063638 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.063730 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.063892 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36583,"prepare.queue_time_us":362091,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"thread_start_us":525972,"threads_started":2}
I20260927 23:31:22.987167 3687 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.083s
I20260927 23:31:22.934180 3910 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:23.085476 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 205347us
W20260927 23:31:23.089993 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 47744.1us
I20260927 23:31:23.104907 2257 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:23.105367 1628 leader_election.cc:334] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.198s (SENT)
I20260927 23:31:23.105970 1628 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:23.110164 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.075466 3707 consensus_queue.cc:1225] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer 0fe98f486bab4d0b9c421c145d83f1ce log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260927 23:31:23.111004 1732 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:23.114264 1654 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:23.123150 1662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s user 0.008s sys 0.017s
W20260927 23:31:23.140429 1662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.010s sys 0.018s
W20260927 23:31:23.139871 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 254262us
I20260927 23:31:23.141597 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:23.139681 1628 leader_election.cc:334] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.238s (SENT)
W20260927 23:31:23.144052 1729 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.011s sys 0.018s
W20260927 23:31:23.145684 1729 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.598s user 0.011s sys 0.018s
I20260927 23:31:23.166471 1657 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:23.116623 3875 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.180384 1628 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:23.187813 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.141016 3776 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:23.193609 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 266808us
I20260927 23:31:23.174630 3910 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:23.087435 3841 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:23.210098 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 306257us
I20260927 23:31:23.182230 3942 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.180433 3897 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.212134 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:23.218472 1628 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:23.200775 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 486659us
W20260927 23:31:23.222668 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 9c719671f35640549c1b944112b3fc03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260927 23:31:23.203965 1702 leader_election.cc:334] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.671s (SENT)
I20260927 23:31:23.196748 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.199369 3875 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.227795 1700 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:23.225729 1702 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:23.231560 1702 leader_election.cc:334] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.225s (SENT)
W20260927 23:31:23.238158 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 133337us
W20260927 23:31:23.224397 3957 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
I20260927 23:31:23.241569 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:23.219462 3946 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.251595 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:23.263405 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.253209 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1347) took 1568 ms. Trace:
I20260927 23:31:23.209704 3776 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260927 23:31:23.222227 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1362) took 1097 ms. Trace:
I20260927 23:31:23.225596 3841 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:23.251017 1702 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:23.210014 3689 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
W20260927 23:31:23.258018 2287 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 398) took 5003 ms (client timeout 2200 ms). Trace:
I20260927 23:31:23.210367 3693 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.266542 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:23.264248 1728 rpcz_store.cc:276] 0927 23:31:21.684368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.124868 (+440500us) service_pool.cc:227] Handling call
0927 23:31:22.163355 (+ 38487us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.435518 (+272163us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.252979 (+817461us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b540011c954. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:23.253152 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":619624832}
I20260927 23:31:23.265916 1730 rpcz_store.cc:276] 0927 23:31:22.124659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.622701 (+498042us) service_pool.cc:227] Handling call
0927 23:31:22.648372 (+ 25671us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.648598 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:23.035726 (+387128us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.221869 (+186143us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:23.221924 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:23.221974 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.222038 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.222173 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1423,"prepare.queue_time_us":252498,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":27,"raft.queue_time_us":3470,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":10752,"thread_start_us":571596,"threads_started":2}
I20260927 23:31:23.289736 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:23.290567 1579 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:23.216364 3924 consensus_queue.cc:1035] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260927 23:31:23.286722 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.289456 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 239) took 1038 ms. Trace:
I20260927 23:31:23.290518 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.215934 3952 consensus_queue.cc:1035] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260927 23:31:23.254866 3825 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:23.215762 3897 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.216511 3942 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.298439 1658 rpcz_store.cc:276] 0927 23:31:22.250927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.258395 (+ 7468us) service_pool.cc:227] Handling call
0927 23:31:23.280559 (+1022164us) spinlock_profiling.cc:243] Waited 79.8 ms on lock 0x7b54002f3ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:23.289390 (+ 8831us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183485312}
I20260927 23:31:23.325840 1730 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:23.229908 3972 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:23.332768 3981 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.333463 3988 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:23.332772 3980 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:23.258246 3901 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:23.265779 3977 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:23.339759 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.340445 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1361) took 1276 ms. Trace:
I20260927 23:31:23.364251 1729 rpcz_store.cc:276] 0927 23:31:22.064015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.489107 (+425092us) service_pool.cc:227] Handling call
0927 23:31:22.489338 (+ 231us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.489515 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:23.146986 (+657471us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.338002 (+191016us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:23.338054 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:23.338100 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.338167 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.338312 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63749,"prepare.queue_time_us":384016,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":22022,"thread_start_us":789689,"threads_started":2}
I20260927 23:31:23.252779 3873 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260927 23:31:23.368940 3901 raft_consensus.cc:686] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:23.368991 3747 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.369786 3873 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:23.370391 3873 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:23.346169 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.370594 3696 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:23.370990 1732 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:23.371167 3696 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:23.371228 3873 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.372138 3980 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.373059 3873 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:23.350656 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 205906us
I20260927 23:31:23.297497 3642 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:23.258033 3875 consensus_queue.cc:1035] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.084s
I20260927 23:31:23.360716 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.363729 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:23.380079 1733 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:23.340945 3822 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:23.288143 2287 rpcz_store.cc:269] 0927 23:31:18.234830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:18.243939 (+ 9109us) service_pool.cc:227] Handling call
0927 23:31:23.227907 (+4983968us) spinlock_profiling.cc:243] Waited 3.53 s on lock 0x7b54005cdd54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:23.238518 (+ 10611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8116859520}
I20260927 23:31:23.384812 1658 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 23:31:23.368829 1729 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:23.369938 3901 consensus_queue.cc:237] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.361217 3986 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:23.352917 3638 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.374102 3980 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.359833 3925 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.409786 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:23.382670 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:23.406102 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 139651us
I20260927 23:31:23.381126 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.219710 3975 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
W20260927 23:31:23.401510 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 137703us
I20260927 23:31:23.396798 3904 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.366791 3981 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.414434 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:23.383360 1730 raft_consensus.cc:2379] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:23.421779 1730 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:23.384752 3689 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.407866 3942 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:23.428894 1628 leader_election.cc:334] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.759s (SENT)
I20260927 23:31:23.387267 3761 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.397146 3665 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.427459 1730 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:23.419279 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:23.430294 2257 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:23.402221 3966 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.250s user 0.001s sys 0.000s
W20260927 23:31:23.438596 3985 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260927 23:31:23.424587 3925 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260927 23:31:23.441706 3757 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.389568 3687 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20260927 23:31:23.445405 2289 raft_consensus.cc:2396] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 23:31:23.478082 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:23.457702 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 67457.3us
I20260927 23:31:23.458590 1732 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:23.545197 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.546972 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.547619 1729 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:23.499476 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:23.497780 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:23.551460 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.552773 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:23.553705 1730 raft_consensus.cc:2396] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:23.554661 1659 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:23.555320 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:23.556137 1655 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:23.459939 3970 log.cc:924] Time spent T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.246s user 0.001s sys 0.001s
I20260927 23:31:23.453198 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:23.579249 3973 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.272s user 0.001s sys 0.000s
I20260927 23:31:23.580454 2288 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260927 23:31:23.582629 3965 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.324s user 0.001s sys 0.001s
W20260927 23:31:23.583673 3997 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.180s user 0.000s sys 0.000s
I20260927 23:31:23.584358 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:23.584810 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 334) took 1184 ms. Trace:
I20260927 23:31:23.585353 2292 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
W20260927 23:31:23.586086 3984 log.cc:924] Time spent T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.238s user 0.001s sys 0.000s
I20260927 23:31:23.444140 3981 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:23.464938 3866 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.587154 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1702) took 1153 ms. Trace:
I20260927 23:31:23.592221 1662 rpcz_store.cc:276] 0927 23:31:22.433459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.438742 (+ 5283us) service_pool.cc:227] Handling call
0927 23:31:22.457023 (+ 18281us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.457267 (+ 244us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:22.457298 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:22.457797 (+ 499us) log.cc:842] Serialized 26 byte log entry
0927 23:31:23.142786 (+684989us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:23.142836 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.142924 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:23.586973 (+444049us) raft_consensus.cc:1671] finished
0927 23:31:23.586990 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.587103 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":684600,"threads_started":1,"wal-append.queue_time_us":772151}
I20260927 23:31:23.585229 1580 rpcz_store.cc:276] 0927 23:31:22.400444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.400634 (+ 190us) service_pool.cc:227] Handling call
0927 23:31:22.404485 (+ 3851us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.425321 (+ 20836us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:22.425366 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:22.445283 (+ 19917us) log.cc:842] Serialized 26 byte log entry
0927 23:31:22.903180 (+457897us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:22.903218 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.903280 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:23.584620 (+681340us) raft_consensus.cc:1671] finished
0927 23:31:23.584639 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.584756 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":320,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2636,"thread_start_us":381891,"threads_started":1,"wal-append.queue_time_us":585578}
I20260927 23:31:23.619793 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:23.620726 1655 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:23.556981 3993 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:23.495450 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:23.635524 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:23.434499 3689 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.453312 3761 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.642616 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:23.573319 2285 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:23.598718 3951 log.cc:924] Time spent T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260927 23:31:23.647894 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:23.656181 1655 raft_consensus.cc:2379] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 8: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:23.461616 3665 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:23.634220 3985 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260927 23:31:23.656191 1627 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:23.587423 3525 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
I20260927 23:31:23.666512 1700 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:23.564713 3642 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260927 23:31:23.630718 3875 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:23.673900 3990 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
I20260927 23:31:23.642004 3993 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:23.673133 1729 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
W20260927 23:31:23.701409 3989 log.cc:924] Time spent T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260927 23:31:23.702822 3761 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.710618 1627 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:23.711242 1655 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:23.648003 3897 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:23.714751 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 209171us
I20260927 23:31:23.716354 1729 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:23.634733 3981 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.042s
I20260927 23:31:23.703773 3689 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:23.719007 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 487438us
I20260927 23:31:23.632055 3696 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.675351 3952 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:23.732167 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:23.708945 3947 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.012s sys 0.015s
I20260927 23:31:23.704237 3988 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:23.747139 2258 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:23.707979 2290 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 23:31:23.748571 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 182002us
I20260927 23:31:23.713099 3993 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:23.752758 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:23.741380 3947 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.987s user 0.012s sys 0.015s
I20260927 23:31:23.748694 3825 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:23.713268 3875 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.761487 3757 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.787961 4009 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:23.768668 3840 consensus_meta.cc:287] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.176s user 0.001s sys 0.011s
I20260927 23:31:23.753938 3952 raft_consensus.cc:686] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:23.800302 3763 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.795749 3980 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:23.745054 3758 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
W20260927 23:31:23.801874 1582 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1441) took 3886 ms (client timeout 4700 ms). Trace:
W20260927 23:31:23.864588 1582 rpcz_store.cc:269] 0927 23:31:19.915275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.605524 (+690249us) service_pool.cc:227] Handling call
0927 23:31:23.801662 (+3196138us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400466254. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:23.801814 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3832928512}
I20260927 23:31:23.769318 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:23.840843 4004 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.087s
W20260927 23:31:23.812202 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 52084us
I20260927 23:31:23.840142 4006 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.103s
I20260927 23:31:23.849206 3665 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.844244 3840 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:23.865573 3910 consensus_meta.cc:287] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.664s user 0.005s sys 0.005s
I20260927 23:31:23.789985 3825 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.849907 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1350) took 2103 ms. Trace:
W20260927 23:31:23.856777 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 47943.2us
I20260927 23:31:23.847765 3952 consensus_queue.cc:237] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:23.801883 3994 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:23.901993 1663 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260927 23:31:23.852418 3875 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.858386 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 416) took 3188 ms. Trace:
W20260927 23:31:23.857963 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 121605us
I20260927 23:31:23.903687 1629 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:23.858932 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 134503us
I20260927 23:31:23.859644 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:23.841185 3974 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
W20260927 23:31:23.861348 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 114328us
I20260927 23:31:23.866000 1733 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:23.769845 3947 raft_consensus.cc:686] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:23.809194 3687 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
W20260927 23:31:23.891157 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1329) took 1607 ms (client timeout 1817 ms). Trace:
I20260927 23:31:23.847766 3977 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:23.898849 1731 rpcz_store.cc:276] 0927 23:31:21.746634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.394015 (+647381us) service_pool.cc:227] Handling call
0927 23:31:22.424615 (+ 30600us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.437322 (+ 12707us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:22.594190 (+156868us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.191219 (+597029us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:23.197224 (+ 6005us) log.cc:842] Serialized 26 byte log entry
0927 23:31:23.197365 (+ 141us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:23.197394 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.197459 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:23.849688 (+652229us) raft_consensus.cc:1671] finished
0927 23:31:23.849723 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.849849 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20070,"prepare.queue_time_us":348929,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":3616,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":25254,"thread_start_us":624062,"threads_started":2,"wal-append.queue_time_us":1110428}
W20260927 23:31:23.867951 4011 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.164s user 0.002s sys 0.000s
I20260927 23:31:23.905294 1629 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:23.918304 1730 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:23.919210 2291 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 4. Ops: []
I20260927 23:31:23.824759 4009 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.912878 3910 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:23.954155 1702 leader_election.cc:334] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.129s (SENT)
I20260927 23:31:23.919188 3875 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:23.922662 1581 rpcz_store.cc:269] 0927 23:31:22.283498 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.312980 (+ 29482us) service_pool.cc:227] Handling call
0927 23:31:23.890911 (+1577931us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b5400242554. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:23.891071 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216333056}
W20260927 23:31:23.999548 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 94341.4us
I20260927 23:31:23.868357 3980 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:23.903059 2293 rpcz_store.cc:276] 0927 23:31:20.602916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:20.603079 (+ 163us) service_pool.cc:227] Handling call
0927 23:31:21.722590 (+1119511us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:21.722765 (+ 175us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:21.722955 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:21.914414 (+191459us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.088496 (+1174082us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b18003dd740. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f657b
0927 23:31:23.590445 (+501949us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b18003dd740. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db44155 00007f9754444167 00007f9750a79807 00007f9750a88f0f
0927 23:31:23.590620 (+ 175us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.788307 (+197687us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:23.788467 (+ 160us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:23.790521 (+ 2054us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.790697 (+ 176us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:23.790900 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":509,"prepare.queue_time_us":488922,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":227661,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":18995,"spinlock_wait_cycles":43485696,"thread_start_us":535871,"threads_started":4,"wal-append.queue_time_us":122080,"wal-append.run_cpu_time_us":1970,"wal-append.run_wall_time_us":1187663}
W20260927 23:31:24.008922 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 111221us
W20260927 23:31:24.001091 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 82332.2us
I20260927 23:31:24.002019 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:23.969218 3974 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:24.013468 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.020491 1657 raft_consensus.cc:2396] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:23.972374 3763 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:23.972992 3948 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:23.977344 3982 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:24.022557 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 68448.9us
W20260927 23:31:24.039062 1702 leader_election.cc:334] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.909s (SENT)
I20260927 23:31:24.039572 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:24.025336 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:24.040795 1731 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:24.003208 3995 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.595s user 0.001s sys 0.002s
I20260927 23:31:23.999840 3696 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:23.992839 4012 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:24.043805 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:24.045681 1702 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.926s (SENT)
I20260927 23:31:24.047113 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:24.049204 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:24.049998 1731 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:24.051164 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.002905 3972 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:24.000043 4025 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:24.062872 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.063373 1730 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:24.012813 3758 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:24.065053 1662 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 7: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:24.067049 1702 leader_election.cc:334] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.221s (SENT)
I20260927 23:31:24.060379 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.068634 1627 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:24.079267 1731 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:24.035306 4006 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:24.090588 1702 leader_election.cc:334] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.600s (SENT)
I20260927 23:31:24.091219 1702 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:24.011199 3910 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.043560 3763 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.048146 3982 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.061383 4012 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.070185 3972 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.082351 4025 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:24.134408 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 51802us
W20260927 23:31:24.140630 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 76220.1us
W20260927 23:31:24.093798 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 182269us
I20260927 23:31:24.069558 3924 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:24.165738 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75189.1us
I20260927 23:31:24.045537 3948 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.053360 3897 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260927 23:31:24.260464 3975 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:23.975740 3431 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.169055 4016 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:24.170534 1731 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 23:31:24.171314 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1368) took 1340 ms. Trace:
W20260927 23:31:24.319304 4008 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.424s user 0.006s sys 0.000s
W20260927 23:31:24.175129 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 79720.7us
I20260927 23:31:24.244366 1657 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:24.059185 3947 consensus_queue.cc:237] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.081867 3745 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:24.154719 3974 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.311192 1627 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:24.322346 1582 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:24.062608 3866 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:24.324175 1702 leader_election.cc:334] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.021s (SENT)
I20260927 23:31:24.264266 4013 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.348834 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:24.352535 1728 rpcz_store.cc:276] 0927 23:31:22.831196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.424761 (+593565us) service_pool.cc:227] Handling call
0927 23:31:23.455245 (+ 30484us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:23.456271 (+ 1026us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:23.456298 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:23.466719 (+ 10421us) log.cc:842] Serialized 26 byte log entry
0927 23:31:23.995091 (+528372us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:23.995154 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.995232 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:24.171140 (+175908us) raft_consensus.cc:1671] finished
0927 23:31:24.171157 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:24.171266 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":662,"raft.queue_time_us":4756,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":49168,"thread_start_us":363203,"threads_started":1,"wal-append.queue_time_us":592291}
I20260927 23:31:24.172519 2285 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:24.315274 3924 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:24.415450 4018 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.358s user 0.003s sys 0.003s
W20260927 23:31:24.356455 3841 consensus_meta.cc:287] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.070s user 0.009s sys 0.003s
I20260927 23:31:24.431797 1657 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
W20260927 23:31:24.434104 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 109959us
W20260927 23:31:24.434507 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 123384us
I20260927 23:31:24.326623 3822 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:24.437929 1581 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:24.349076 4004 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:24.356238 3901 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.430223 4016 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.347365 4019 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.381413 3975 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:24.451112 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1596) took 3017 ms. Trace:
I20260927 23:31:24.495290 3763 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.447782 3757 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:24.495702 1736 rpcz_store.cc:276] 0927 23:31:21.433770 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:21.516009 (+ 82239us) service_pool.cc:227] Handling call
0927 23:31:21.516287 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.478630 (+962343us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.450878 (+1972248us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54002ee0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:24.451040 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2172515072}
I20260927 23:31:24.447958 3924 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:24.499224 3910 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:24.495358 3975 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.499823 3910 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:24.502044 3972 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:24.451164 1702 leader_election.cc:334] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.979s (SENT)
W20260927 23:31:24.484733 1628 leader_election.cc:334] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.957s (SENT)
I20260927 23:31:24.502655 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:24.429569 3825 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.503373 3866 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:24.503545 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 52417.5us
I20260927 23:31:24.450812 3758 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:24.490463 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1359) took 2532 ms. Trace:
I20260927 23:31:24.414479 3840 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:24.508900 1702 leader_election.cc:334] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 4.700s (SENT)
W20260927 23:31:24.511615 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 59354.3us
I20260927 23:31:24.512094 1657 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 23:31:24.396751 3687 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.122s
I20260927 23:31:24.452768 2285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 429) took 1032 ms. Trace:
I20260927 23:31:24.514390 1735 rpcz_store.cc:276] 0927 23:31:21.958194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.487365 (+529171us) service_pool.cc:227] Handling call
0927 23:31:22.519011 (+ 31646us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.585322 (+ 66311us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:22.907806 (+322484us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.195445 (+287639us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:23.199478 (+ 4033us) log.cc:842] Serialized 26 byte log entry
0927 23:31:23.226684 (+ 27206us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:23.226738 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.463540 (+1236802us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b54001ddb54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:24.485073 (+ 21533us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:24.485114 (+ 41us) raft_consensus.cc:1671] finished
0927 23:31:24.489817 (+ 4703us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:24.490360 (+ 543us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5858,"prepare.queue_time_us":315494,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":6514,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":1221,"spinlock_wait_cycles":120203904,"thread_start_us":499666,"threads_started":2,"wal-append.queue_time_us":422143}
I20260927 23:31:24.527558 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:24.527558 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 84483.9us
I20260927 23:31:24.441967 3971 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:24.522344 3683 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.480810 3822 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.446553 4013 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.550474 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:24.556864 3822 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.505694 3901 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.520684 4009 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
W20260927 23:31:24.563903 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 55036.2us
I20260927 23:31:24.545203 3825 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:24.542727 3841 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.515357 3757 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.487229 4004 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.528123 2285 rpcz_store.cc:276] 0927 23:31:23.416049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.420423 (+ 4374us) service_pool.cc:227] Handling call
0927 23:31:24.447902 (+1027479us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:24.580845 3955 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.813s user 0.010s sys 0.000s
I20260927 23:31:24.581252 3758 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:24.582253 3758 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.583842 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1714) took 1554 ms. Trace:
I20260927 23:31:24.584231 1654 rpcz_store.cc:276] 0927 23:31:23.029700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.029896 (+ 196us) service_pool.cc:227] Handling call
0927 23:31:23.066085 (+ 36189us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:23.115478 (+ 49393us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.583622 (+1468144us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b54000db6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:24.583791 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110306176}
I20260927 23:31:24.585318 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.585968 1654 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:24.587842 1734 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:24.595382 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1614) took 2859 ms. Trace:
I20260927 23:31:24.516422 4019 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.595883 1734 rpcz_store.cc:276] 0927 23:31:21.735905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.391005 (+655100us) service_pool.cc:227] Handling call
0927 23:31:22.412986 (+ 21981us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:24.587653 (+2174667us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b5400077720. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f97509ff730 00007f9750a71300 00007f9750a72371 00007f9750a6eedf 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc
0927 23:31:24.595129 (+ 7476us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.595320 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36124544}
I20260927 23:31:24.600066 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:24.604070 4004 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:24.586953 4009 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260927 23:31:24.605587 4013 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:24.608700 4033 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260927 23:31:24.605748 3723 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 23:31:24.594807 4021 log.cc:924] Time spent T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260927 23:31:24.539248 3971 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:24.614600 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 89771.9us
I20260927 23:31:24.517563 4016 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.597179 1581 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:24.616905 4048 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260927 23:31:24.598707 4012 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.611119 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:24.576645 3841 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.561668 3972 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260927 23:31:24.600656 3982 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.567821 4049 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:24.606918 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:24.566371 3683 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:24.767813 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 201487us
W20260927 23:31:24.687829 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 115337us
W20260927 23:31:24.789044 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 216826us
I20260927 23:31:24.789963 1629 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:24.740254 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 116109us
W20260927 23:31:24.625164 4023 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20260927 23:31:24.611967 3948 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:24.813208 1654 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:24.626984 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:24.819128 1551 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:24.750195 4007 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.305s user 0.000s sys 0.001s
I20260927 23:31:24.821108 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:24.821507 2259 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:24.616016 3910 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.189s
I20260927 23:31:24.825008 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:24.827894 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.827699 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:24.830341 1736 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:24.830654 1734 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:24.760587 4025 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:24.838121 1582 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 23:31:24.836040 1728 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:24.818449 3689 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:24.862135 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:24.863816 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.864555 1662 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:24.865520 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.769948 4016 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:24.868973 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.870393 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:24.860620 2285 raft_consensus.cc:2396] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 23:31:24.879118 4019 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.820684 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:24.861867 4028 log.cc:924] Time spent T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.335s user 0.000s sys 0.001s
I20260927 23:31:24.805058 3924 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.890743 1662 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 23:31:24.898471 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 430) took 1473 ms. Trace:
I20260927 23:31:24.867308 3982 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:24.901032 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:24.890619 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 52646.1us
I20260927 23:31:24.901736 1736 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:24.898854 2291 rpcz_store.cc:276] 0927 23:31:23.424996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.425182 (+ 186us) service_pool.cc:227] Handling call
0927 23:31:23.453802 (+ 28620us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:23.999787 (+545985us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.898146 (+898359us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54004f1354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:24.898308 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069902080}
I20260927 23:31:24.863488 4058 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:24.867162 3757 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:24.829749 3948 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20260927 23:31:24.896984 4039 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260927 23:31:24.789453 4049 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:24.835569 4059 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.087s
I20260927 23:31:24.903903 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:24.905526 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:24.983297 3732 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260927 23:31:24.983521 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 79723.2us
I20260927 23:31:24.983731 3732 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:24.984036 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 163390us
W20260927 23:31:24.950582 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 83334.6us
I20260927 23:31:24.907370 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:24.912164 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:25.019488 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:25.013236 1627 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:24.902420 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:24.891908 3689 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:24.929083 1734 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 23:31:24.968436 3907 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:24.901153 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:25.027096 4061 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20260927 23:31:25.026271 3907 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:25.027765 3983 log.cc:924] Time spent T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.590s user 0.000s sys 0.001s
W20260927 23:31:25.029188 3993 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 1.256s user 0.006s sys 0.006s
I20260927 23:31:24.912398 3974 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:25.029572 3993 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:24.999418 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 94711.4us
I20260927 23:31:25.030735 4019 raft_consensus.cc:2741] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 23:31:25.030992 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1357) took 3107 ms. Trace:
W20260927 23:31:25.031622 4067 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
W20260927 23:31:25.022917 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 120584us
I20260927 23:31:25.032922 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:25.026234 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 503664us
W20260927 23:31:25.036077 1702 leader_election.cc:334] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.836s (SENT)
I20260927 23:31:25.020155 1728 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:25.036525 1702 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:25.021526 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:25.024173 3758 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:25.020826 4059 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:25.042668 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1376) took 1546 ms. Trace:
I20260927 23:31:25.031404 1737 rpcz_store.cc:276] 0927 23:31:21.923865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.450123 (+526258us) service_pool.cc:227] Handling call
0927 23:31:22.459557 (+ 9434us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.531040 (+ 71483us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:22.720812 (+189772us) log.cc:842] Serialized 14 byte log entry
0927 23:31:23.140979 (+420167us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:23.141393 (+ 414us) log.cc:842] Serialized 26 byte log entry
0927 23:31:23.141499 (+ 106us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:23.141527 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:23.141583 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.027253 (+1885670us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:25.030720 (+ 3467us) raft_consensus.cc:1671] finished
0927 23:31:25.030815 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.030938 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42753,"prepare.queue_time_us":369477,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":51386,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":39412,"spinlock_wait_cycles":30877056,"thread_start_us":495830,"threads_started":2,"wal-append.queue_time_us":631544}
W20260927 23:31:25.027225 4062 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260927 23:31:25.042431 3758 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:25.053241 3758 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:25.054726 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:25.055518 1737 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:25.043056 1732 rpcz_store.cc:276] 0927 23:31:23.496474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.566855 (+ 70381us) service_pool.cc:227] Handling call
0927 23:31:23.567253 (+ 398us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:23.598066 (+ 30813us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:23.598092 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:23.598643 (+ 551us) log.cc:842] Serialized 52 byte log entry
0927 23:31:23.898394 (+299751us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:23.921136 (+ 22742us) log.cc:842] Serialized 14 byte log entry
0927 23:31:24.001107 (+ 79971us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.001178 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.042483 (+1041305us) raft_consensus.cc:1671] finished
0927 23:31:25.042499 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.042615 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90845,"prepare.queue_time_us":22534,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":120115,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":39419,"thread_start_us":245184,"threads_started":1,"wal-append.queue_time_us":421173}
I20260927 23:31:25.033452 3974 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.023782 3924 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:24.891734 4044 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:24.920946 4058 raft_consensus.cc:686] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:25.025241 3689 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:24.988063 3910 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:24.904052 3982 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:25.020619 3732 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.078748 3924 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260927 23:31:25.034005 4022 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260927 23:31:24.902217 4016 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:25.042011 3591 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
I20260927 23:31:25.050454 4004 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:25.051186 3986 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:25.051036 4019 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:25.052050 3117 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:25.052484 4071 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:24.929730 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.056787 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1385) took 1045 ms. Trace:
I20260927 23:31:24.928303 3971 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.062332 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1536) took 2477 ms. Trace:
W20260927 23:31:25.038064 1628 leader_election.cc:334] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.023s (SENT)
W20260927 23:31:25.105809 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 67788.7us
I20260927 23:31:25.105377 1656 rpcz_store.cc:276] 0927 23:31:22.585395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.585498 (+ 103us) service_pool.cc:227] Handling call
0927 23:31:22.640101 (+ 54603us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:22.840342 (+200241us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:22.840390 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:22.840450 (+ 60us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:22.840502 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.062050 (+2221548us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54000bc2d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:25.062095 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.062248 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459823360}
W20260927 23:31:25.106731 1628 leader_election.cc:334] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.026s (SENT)
W20260927 23:31:25.107429 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 85973.9us
I20260927 23:31:25.084343 3993 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.071411 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:24.985839 3683 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.034330 3907 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.068163 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:25.082592 1702 leader_election.cc:334] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.881s (SENT)
W20260927 23:31:25.085839 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65637.3us
I20260927 23:31:25.059224 4059 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.088727 3841 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.095325 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.096071 1654 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:25.123998 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.124864 1656 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.096163 3758 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.100776 2292 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:25.078598 3910 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:25.102294 1729 rpcz_store.cc:276] 0927 23:31:24.011454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.312887 (+301433us) service_pool.cc:227] Handling call
0927 23:31:24.314305 (+ 1418us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:24.687618 (+373313us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:24.687662 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:24.769986 (+ 82324us) log.cc:842] Serialized 26 byte log entry
0927 23:31:24.997021 (+227035us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:24.997070 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.997170 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.056579 (+ 59409us) raft_consensus.cc:1671] finished
0927 23:31:25.056598 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.056724 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8659,"raft.queue_time_us":42420,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":56321,"thread_start_us":544730,"threads_started":2,"wal-append.queue_time_us":245492}
I20260927 23:31:25.081321 3982 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.040746 4012 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260927 23:31:25.107707 1628 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.040s (SENT)
W20260927 23:31:25.135000 1628 leader_election.cc:334] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.149s (SENT)
I20260927 23:31:25.135689 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:25.117063 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:25.119581 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:25.137543 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 54989.8us
I20260927 23:31:25.088379 3986 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:25.119334 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 80232.7us
W20260927 23:31:25.122632 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 51349us
I20260927 23:31:25.080348 4058 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.088169 4004 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.106572 4019 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:25.142659 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.086201 4016 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.145277 4049 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.146857 3758 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.146764 3910 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.139281 4012 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:25.148494 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1383) took 1284 ms. Trace:
I20260927 23:31:25.117499 4071 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:25.150823 3924 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:25.151172 3687 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:25.151541 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.131897 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.196105 1662 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
W20260927 23:31:25.154958 1628 leader_election.cc:334] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.202s (SENT)
I20260927 23:31:25.126974 3683 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:25.172869 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:25.198783 1737 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:25.175887 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:25.178861 1730 rpcz_store.cc:276] 0927 23:31:23.864058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.171596 (+307538us) service_pool.cc:227] Handling call
0927 23:31:24.172113 (+ 517us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:24.172983 (+ 870us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:24.173626 (+ 643us) log.cc:842] Serialized 14 byte log entry
0927 23:31:24.259941 (+ 86315us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:24.265508 (+ 5567us) log.cc:842] Serialized 52 byte log entry
0927 23:31:24.265701 (+ 193us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:24.265999 (+ 298us) log.cc:842] Serialized 14 byte log entry
0927 23:31:24.266231 (+ 232us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:24.266311 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.142434 (+876123us) raft_consensus.cc:1671] finished
0927 23:31:25.142450 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.148428 (+ 5978us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":278,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":4039,"raft.queue_time_us":12949,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":11187,"spinlock_wait_cycles":15403264,"thread_start_us":74129,"threads_started":1,"wal-append.queue_time_us":266122}
I20260927 23:31:25.188613 1656 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:25.139464 3117 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:25.189793 2258 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:25.190601 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 73844.7us
I20260927 23:31:25.214969 1552 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.162730 4044 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:25.173360 4019 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:25.217613 1551 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:25.211967 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:25.218782 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 63867.9us
W20260927 23:31:25.219609 1628 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.245s (SENT) [suppressed 55 similar messages]
I20260927 23:31:25.145463 4059 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.224040 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:25.221972 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:25.226768 1728 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:25.187465 4012 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:25.193861 3687 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:25.218570 3910 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:25.238551 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.216216 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 493) took 1754 ms. Trace:
I20260927 23:31:25.221676 4044 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:25.225522 4019 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.247721 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:25.254030 1730 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:25.233898 4059 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:25.225172 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.235126 3974 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.269806 1728 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:25.247285 2286 rpcz_store.cc:276] 0927 23:31:23.461396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.463879 (+ 2483us) service_pool.cc:227] Handling call
0927 23:31:25.161126 (+1697247us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0fbce 000000000034aea9 00007f975593ba30 00007f9755a017b4 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:25.161294 (+ 168us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:25.175592 (+ 14298us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:25.178056 (+ 2464us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:25.185838 (+ 7782us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:25.185877 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.189493 (+ 3616us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.214008 (+ 24515us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28380544}
I20260927 23:31:25.270522 3687 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.249248 4022 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:25.248364 3707 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:25.277977 1552 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.210094 4071 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.260414 4044 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.281561 1656 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260927 23:31:25.282032 1656 pending_rounds.cc:76] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 4
I20260927 23:31:25.282301 1656 pending_rounds.cc:97] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.5
I20260927 23:31:25.279491 3687 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:25.280757 4059 consensus_queue.cc:1035] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260927 23:31:25.234848 3732 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.285354 4059 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:25.287253 1656 raft_consensus.cc:2894] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Aborting config change with OpId 2.5: Aborted: Op aborted by new leader
I20260927 23:31:25.287904 1656 raft_consensus.cc:2866] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:25.288848 1730 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:25.276389 4080 consensus_queue.cc:1159] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a3317e8da742d9aefe4c251abd4566" responder_term: 6 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 23:31:25.285421 3732 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.289003 4077 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:25.312114 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46178 (request call id 890) took 9945 ms. Trace:
I20260927 23:31:25.281252 3707 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.317312 3707 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.318234 3776 raft_consensus.cc:3032] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:25.318380 3901 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:25.318739 3776 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:25.325402 4077 raft_consensus.cc:686] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:25.313364 1656 rpcz_store.cc:276] 0927 23:31:15.372329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:15.834807 (+462478us) service_pool.cc:227] Handling call
0927 23:31:17.005153 (+1170346us) spinlock_profiling.cc:243] Waited 9.02 ms on lock 0x7b4c00049100. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974ee32210 00007f974ee42fb2 00007f974ee09ac5 00007f974ee3071d 00007f974ee30383 00007f97509e8a79 00007f97509e88d4 00007f9750a4ee6e 00007f9750a6aacc 00007f9750a6bada 00007f9750a6ac8b
0927 23:31:17.007081 (+ 1928us) log.cc:842] Serialized 480 byte log entry
0927 23:31:25.310088 (+8303007us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145,"raft.run_cpu_time_us":706,"raft.run_wall_time_us":876,"spinlock_wait_cycles":20750720,"thread_start_us":87573,"threads_started":1,"wal-append.queue_time_us":166509,"wal-append.run_cpu_time_us":3603,"wal-append.run_wall_time_us":1228848}
I20260927 23:31:25.327807 3776 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.328434 3707 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:25.328928 3707 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.331796 4071 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.333283 3707 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.328738 4077 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.343482 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.319787 4004 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:25.366655 4012 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:25.369001 3976 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 2.153s user 0.003s sys 0.001s
I20260927 23:31:25.345420 4016 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.389703 4044 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.343752 3982 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.406276 1582 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:25.409195 4044 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:25.409902 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 336) took 2960 ms. Trace:
I20260927 23:31:25.410306 1588 rpcz_store.cc:276] 0927 23:31:22.449925 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.461611 (+ 11686us) service_pool.cc:227] Handling call
0927 23:31:22.498842 (+ 37231us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:22.499268 (+ 426us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:22.499311 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:22.512848 (+ 13537us) log.cc:842] Serialized 26 byte log entry
0927 23:31:22.960014 (+447166us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:22.960061 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:22.960176 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.404572 (+2444396us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b24004013a8. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974d99fcf4 00007f974d99ddfb 00007f974da2c559 00007f974da2c1e5 00007f974da2b740 00007f975071e578 00007f975071ecd8 00007f97506f66e8 00007f97506f1aea 00007f97506f0f36
0927 23:31:25.406059 (+ 1487us) raft_consensus.cc:1671] finished
0927 23:31:25.406082 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.409838 (+ 3756us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6183,"raft.queue_time_us":5434,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":18454,"spinlock_wait_cycles":24634240,"thread_start_us":378359,"threads_started":1,"wal-append.queue_time_us":670918}
I20260927 23:31:25.377542 3986 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.405997 3982 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:25.437893 4026 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.914s user 0.001s sys 0.001s
I20260927 23:31:25.452306 4019 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.454701 4012 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.458993 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.459800 1654 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.461570 3986 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.463086 4019 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:25.473520 1655 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.478648 3980 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:25.487346 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1387) took 1176 ms. Trace:
I20260927 23:31:25.487900 1731 rpcz_store.cc:276] 0927 23:31:24.311053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.356908 (+ 45855us) service_pool.cc:227] Handling call
0927 23:31:24.357270 (+ 362us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:24.444070 (+ 86800us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:24.444104 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:24.444143 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:24.444183 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.487053 (+1042870us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b54002edbd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:25.487099 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.487257 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199186816}
I20260927 23:31:25.498405 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1539) took 2248 ms. Trace:
I20260927 23:31:25.498896 1660 rpcz_store.cc:276] 0927 23:31:23.249861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.250926 (+ 1065us) service_pool.cc:227] Handling call
0927 23:31:23.251360 (+ 434us) raft_consensus.cc:1457] Updating replica for 3 ops
0927 23:31:23.381879 (+130519us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:23.381909 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0927 23:31:23.387720 (+ 5811us) log.cc:842] Serialized 78 byte log entry
0927 23:31:23.743780 (+356060us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:23.743836 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.489709 (+1745873us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54003afcd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:25.489771 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:25.489831 (+ 60us) raft_consensus.cc:1671] finished
0927 23:31:25.489849 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.498332 (+ 8483us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27818,"raft.queue_time_us":13680,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5659,"spinlock_wait_cycles":262062336,"thread_start_us":333091,"threads_started":1,"wal-append.queue_time_us":850989}
I20260927 23:31:25.508010 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.512625 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1712) took 2720 ms. Trace:
I20260927 23:31:25.513123 1663 rpcz_store.cc:276] 0927 23:31:22.792109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.793731 (+ 1622us) service_pool.cc:227] Handling call
0927 23:31:22.839741 (+ 46010us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:23.902443 (+1062702us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.512321 (+1609878us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540013a1d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:25.512558 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2442289920}
I20260927 23:31:25.517678 3654 raft_consensus.cc:3032] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:25.518118 3654 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:25.518867 3654 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.519768 3654 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.519840 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.520607 1663 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 7
I20260927 23:31:25.519930 1660 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:25.522921 1660 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:25.524969 4019 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:25.525699 4019 consensus_queue.cc:1225] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260927 23:31:25.531548 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1706) took 3018 ms. Trace:
I20260927 23:31:25.532066 1659 rpcz_store.cc:276] 0927 23:31:22.512994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.513160 (+ 166us) service_pool.cc:227] Handling call
0927 23:31:22.539194 (+ 26034us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:23.562649 (+1023455us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.531261 (+1968612us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540013a6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:25.531477 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2334834944}
I20260927 23:31:25.548436 1654 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:25.558763 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1483) took 2929 ms. Trace:
I20260927 23:31:25.559496 1660 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 23:31:25.559310 1585 rpcz_store.cc:276] 0927 23:31:22.629099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:22.986869 (+357770us) service_pool.cc:227] Handling call
0927 23:31:22.992768 (+ 5899us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:23.031717 (+ 38949us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.558511 (+2526794us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b5400467150. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:25.558687 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9080576}
I20260927 23:31:25.562366 1551 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:25.571545 3654 consensus_queue.cc:1046] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260927 23:31:25.605811 1659 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:25.574419 4019 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:25.569535 3696 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:25.582258 3986 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:25.644723 1663 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:25.602854 3980 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:25.601861 3873 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:25.645520 3910 consensus_queue.cc:1046] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:31:25.664506 1659 raft_consensus.cc:1261] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 23:31:25.666225 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.649621 3654 consensus_queue.cc:1225] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260927 23:31:25.809427 3873 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.648842 3986 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.647876 4019 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:25.648090 3696 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.672335 4057 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:25.694016 4048 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:25.712069 3776 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:25.717079 1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 319) took 2337 ms (client timeout 1546 ms). Trace:
W20260927 23:31:25.722954 1628 leader_election.cc:334] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.541s (SENT)
I20260927 23:31:25.814342 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:25.814184 1733 rpcz_store.cc:269] 0927 23:31:23.379572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.380854 (+ 1282us) service_pool.cc:227] Handling call
0927 23:31:25.716871 (+2336017us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54003c3f54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:25.716998 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1044005120}
W20260927 23:31:25.815439 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 92513.8us
W20260927 23:31:25.816148 1628 leader_election.cc:334] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.553s (SENT)
I20260927 23:31:25.815470 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.740818 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.743196 3431 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.454110 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:25.808917 3980 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:25.822849 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:25.784363 2258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.029s sys 0.024s
W20260927 23:31:25.823648 2258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.029s sys 0.024s
W20260927 23:31:25.824412 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 634644us
I20260927 23:31:25.826452 2258 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:25.784618 2257 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.007s sys 0.024s
I20260927 23:31:25.799820 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:25.781328 2290 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.055s sys 0.059s
I20260927 23:31:25.809955 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.809954 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:25.783818 1581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.009s sys 0.043s
W20260927 23:31:25.823534 1737 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.021s sys 0.030s
W20260927 23:31:25.827301 2257 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.008s sys 0.024s
W20260927 23:31:25.794762 1732 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.007s sys 0.007s
I20260927 23:31:25.853323 1659 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
W20260927 23:31:25.853621 1732 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.663s user 0.007s sys 0.007s
W20260927 23:31:25.854025 1661 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.008s sys 0.010s
I20260927 23:31:25.850523 1660 raft_consensus.cc:2379] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
W20260927 23:31:25.850907 1581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.010s sys 0.044s
I20260927 23:31:25.851591 2258 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:25.852237 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 628247us
W20260927 23:31:25.857200 1661 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.008s sys 0.010s
I20260927 23:31:25.858151 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1540) took 2606 ms. Trace:
I20260927 23:31:25.858563 1661 rpcz_store.cc:276] 0927 23:31:23.251873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.252026 (+ 153us) service_pool.cc:227] Handling call
0927 23:31:23.252354 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:23.252564 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:23.281315 (+ 28751us) log.cc:842] Serialized 14 byte log entry
0927 23:31:25.308454 (+2027139us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b18003dcea0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db29345 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb 00007f97506f6063 00007f97506f657b 00007f9750a7988a
0927 23:31:25.857757 (+549303us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:25.857833 (+ 76us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:25.857889 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.857969 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.858094 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27841,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"spinlock_wait_cycles":15005568,"thread_start_us":549049,"threads_started":1}
W20260927 23:31:25.852092 1737 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.021s sys 0.030s
W20260927 23:31:25.850112 1552 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.024s sys 0.027s
W20260927 23:31:25.861790 1552 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.024s sys 0.027s
W20260927 23:31:25.862426 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 584514us
I20260927 23:31:25.863330 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.863894 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:25.818156 4048 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:25.819053 4017 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.021s sys 0.041s
W20260927 23:31:25.826076 4058 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.003s sys 0.015s
W20260927 23:31:25.868227 2293 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.013s sys 0.034s
W20260927 23:31:25.889546 2293 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.013s sys 0.035s
I20260927 23:31:25.876461 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.884730 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:25.891186 1659 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:25.894445 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:25.895195 1731 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:25.850044 2290 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.055s sys 0.059s
I20260927 23:31:25.920341 1663 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 7. Ops: []
I20260927 23:31:25.921365 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:25.929443 1733 raft_consensus.cc:2379] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:25.938755 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:25.940505 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 51145.9us
I20260927 23:31:25.940830 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:25.853221 4066 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:25.946113 1731 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
W20260927 23:31:25.890422 3825 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.019s sys 0.052s
I20260927 23:31:25.968792 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.969046 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.865353 4100 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:25.864388 3975 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:25.893517 4017 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.021s sys 0.042s
I20260927 23:31:25.941414 1735 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
W20260927 23:31:25.891285 4058 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.003s sys 0.016s
I20260927 23:31:25.818078 4057 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.948948 1629 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.919979 3732 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:25.988473 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:25.974264 2293 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:25.974589 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 98227.4us
I20260927 23:31:25.989672 1552 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.978042 4036 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260927 23:31:25.978163 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:25.978484 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:25.978739 4079 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:25.968204 3825 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.732s user 0.019s sys 0.052s
I20260927 23:31:25.893348 3910 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:25.996070 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.993112 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:26.003109 4077 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:26.004329 4057 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:26.005851 2258 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:26.006112 2292 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:26.003340 4069 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:25.976650 3908 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:26.011287 3873 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:26.012327 3696 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.019572 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:25.980479 4066 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:26.014901 1663 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:25.988400 2290 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 500) took 1077 ms. Trace:
I20260927 23:31:26.022271 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:25.980777 4048 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.025667 4066 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.025890 1486 catalog_manager.cc:5495] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: config changed from index 3 to 4, term changed from 1 to 3, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:26.025460 2290 rpcz_store.cc:276] 0927 23:31:24.907631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.930610 (+ 22979us) service_pool.cc:227] Handling call
0927 23:31:24.987395 (+ 56785us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:25.016864 (+ 29469us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:25.210803 (+193939us) log.cc:842] Serialized 14 byte log entry
0927 23:31:25.944560 (+733757us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:25.948381 (+ 3821us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:25.968993 (+ 20612us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.975420 (+ 6427us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:25.985079 (+ 9659us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":190724,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":54167,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4105,"thread_start_us":827138,"threads_started":2}
I20260927 23:31:26.022897 3654 raft_consensus.cc:3032] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:26.030673 3654 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:26.031512 3654 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.023384 3873 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:26.024611 1485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48998 (request call id 232) took 11849 ms (client timeout 15000 ms). Trace:
I20260927 23:31:25.972680 4100 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:25.980167 3975 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:25.973341 4074 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260927 23:31:25.967516 4019 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.053004 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:26.029907 1486 catalog_manager.cc:5495] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 4, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:26.285208 1627 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.037s user 0.015s sys 0.019s
W20260927 23:31:26.354022 1655 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.019s sys 0.039s
I20260927 23:31:26.254220 4057 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.010035 3665 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:26.395491 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 342585us
I20260927 23:31:26.268496 1662 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:26.394881 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:26.409454 1655 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s user 0.023s sys 0.040s
I20260927 23:31:26.343969 2289 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
W20260927 23:31:26.356743 3922 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.006s sys 0.026s
I20260927 23:31:26.407922 1663 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:26.428432 1486 catalog_manager.cc:5495] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 2 to 5, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 5 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:26.430661 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.431249 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.431995 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.432684 1580 raft_consensus.cc:2362] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:26.436285 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.436553 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.436884 1582 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:26.437857 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:26.315078 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.108s user 0.055s sys 0.119s
W20260927 23:31:26.438884 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.232s user 0.057s sys 0.121s
W20260927 23:31:26.372987 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 337285us
I20260927 23:31:26.441212 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.441923 1586 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:26.412384 1627 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.164s user 0.016s sys 0.020s
W20260927 23:31:26.420250 1658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.018s sys 0.042s
W20260927 23:31:26.447091 1658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.977s user 0.018s sys 0.042s
W20260927 23:31:26.356869 3994 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.355s user 0.016s sys 0.073s
I20260927 23:31:26.051667 3862 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:26.032732 3654 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:26.480767 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.033924 3908 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:26.265539 3825 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:26.383428 4049 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.482280 1585 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:26.427613 3910 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:26.456931 3758 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:26.338107 3687 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
W20260927 23:31:26.356495 3900 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.195s user 0.016s sys 0.017s
I20260927 23:31:26.495961 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.560513 1588 raft_consensus.cc:2362] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:26.561873 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.496790 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.107630 3986 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:26.499852 1586 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:26.428541 4002 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.509s
I20260927 23:31:26.267715 4050 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Stepping down as leader of term 5
W20260927 23:31:26.039465 1485 rpcz_store.cc:269] 0927 23:31:14.175072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.175263 (+ 191us) service_pool.cc:227] Handling call
0927 23:31:26.024531 (+11849268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:22.629706 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:22.901486 (+271780us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:22.901539 (+ 53us) write_op.cc:435] Acquired schema lock
0927 23:31:22.901550 (+ 11us) tablet.cc:660] Decoding operations
0927 23:31:23.104092 (+202542us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:23.124390 (+ 20298us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:23.144887 (+ 20497us) tablet.cc:683] Acquiring locks for 11 operations
0927 23:31:23.352359 (+207472us) tablet.cc:699] Row locks acquired
0927 23:31:23.359075 (+ 6716us) write_op.cc:260] PREPARE: finished
0927 23:31:23.382735 (+ 23660us) write_op.cc:270] Start()
0927 23:31:23.408295 (+ 25560us) write_op.cc:276] Timestamp: P: 1790551883375036 usec, L: 0
0927 23:31:23.408339 (+ 44us) op_driver.cc:331] REPLICATION: starting
0927 23:31:23.409052 (+ 713us) log.cc:842] Serialized 4369 byte log entry
0927 23:31:23.444042 (+ 34990us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f974db295fe 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb
0927 23:31:24.580081 (+1136039us) op_driver.cc:444] REPLICATION: finished
0927 23:31:24.896829 (+316748us) write_op.cc:301] APPLY: starting
0927 23:31:24.906781 (+ 9952us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:24.987905 (+ 81124us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:25.008129 (+ 20224us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:25.025397 (+ 17268us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:25.025754 (+ 357us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0927 23:31:25.025821 (+ 67us) write_op.cc:312] APPLY: finished
0927 23:31:25.026212 (+ 391us) log.cc:842] Serialized 83 byte log entry
0927 23:31:25.056993 (+ 30781us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:25.071238 (+ 14245us) write_op.cc:454] Released schema lock
0927 23:31:25.084223 (+ 12985us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:25.121518 (+ 37295us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:25.126683 (+ 5165us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":184655,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":279861,"apply.run_cpu_time_us":5077,"apply.run_wall_time_us":271122,"num_ops":11,"prepare.queue_time_us":227090,"prepare.run_cpu_time_us":37291,"prepare.run_wall_time_us":1060243,"raft.queue_time_us":222641,"raft.run_cpu_time_us":39516,"raft.run_wall_time_us":242986,"replication_time_us":1171638,"spinlock_wait_cycles":6194048,"thread_start_us":639807,"threads_started":4,"wal-append.queue_time_us":294509,"wal-append.run_cpu_time_us":22833,"wal-append.run_wall_time_us":2323413}]]}
I20260927 23:31:26.481272 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.581388 1583 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:26.526263 4016 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.495746 3862 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:26.428179 4035 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.555778 3687 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:26.465202 3922 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s user 0.007s sys 0.027s
I20260927 23:31:26.067451 4040 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:26.037067 4080 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.456s
I20260927 23:31:26.036417 4071 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.457s
W20260927 23:31:26.519027 1702 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.010s sys 0.021s
I20260927 23:31:26.525744 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.076323 3942 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 23:31:26.499279 3994 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.495s user 0.016s sys 0.075s
I20260927 23:31:26.418993 3873 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.525347 3908 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:26.534621 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 150194us
W20260927 23:31:26.618463 4029 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.921s user 0.002s sys 0.040s
I20260927 23:31:26.542747 3654 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:26.077314 3134 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.428s
I20260927 23:31:26.571440 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.655028 1586 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:26.655920 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.656473 1586 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:26.500411 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 60082.5us
I20260927 23:31:26.591126 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.547334 3825 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:26.560102 3900 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.399s user 0.024s sys 0.018s
I20260927 23:31:26.497527 1582 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:26.666097 4029 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.968s user 0.002s sys 0.041s
I20260927 23:31:26.513603 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:26.656774 3947 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s user 0.012s sys 0.038s
I20260927 23:31:26.721019 1581 raft_consensus.cc:2362] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:26.547513 4049 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.641958 1585 raft_consensus.cc:2396] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260927 23:31:26.642406 1702 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.938s user 0.015s sys 0.022s
I20260927 23:31:26.610452 3862 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:26.604624 4050 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:26.561249 3758 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.338393 3980 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.393322 4100 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:26.535387 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 256) took 1912 ms. Trace:
I20260927 23:31:26.751459 1656 rpcz_store.cc:276] 0927 23:31:24.622552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.360634 (+738082us) service_pool.cc:227] Handling call
0927 23:31:25.381441 (+ 20807us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:25.381895 (+ 454us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:25.381927 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:25.386193 (+ 4266us) log.cc:842] Serialized 26 byte log entry
0927 23:31:25.787076 (+400883us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:25.787131 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.787228 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:26.535103 (+747875us) raft_consensus.cc:1671] finished
0927 23:31:26.535130 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:26.535318 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":400353,"threads_started":1,"wal-append.queue_time_us":1008540}
I20260927 23:31:26.657227 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.666181 1583 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:26.612366 4016 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.645670 3994 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:26.651558 3707 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.923s user 0.013s sys 0.029s
W20260927 23:31:26.754773 4099 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260927 23:31:26.715520 3986 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:26.430658 4057 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.658275 4071 consensus_queue.cc:1225] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260927 23:31:26.761958 3707 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s user 0.014s sys 0.031s
I20260927 23:31:26.721279 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.797637 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.603340 1486 catalog_manager.cc:5495] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:26.656491 4040 raft_consensus.cc:686] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:26.724879 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 69074.8us
W20260927 23:31:26.724933 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 1.02122e+06us
I20260927 23:31:26.653471 3908 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.808135 3908 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.737519 3873 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:26.738458 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.809316 1585 raft_consensus.cc:2396] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:26.810079 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.811247 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.811594 4016 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:26.811784 1585 raft_consensus.cc:2362] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:26.812628 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.813314 1585 raft_consensus.cc:2379] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:26.813517 3825 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:26.736651 3947 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s user 0.014s sys 0.043s
I20260927 23:31:26.746889 4080 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:26.722067 4097 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260927 23:31:26.750214 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.755798 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.658566 4009 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
W20260927 23:31:26.762626 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted by new leader
I20260927 23:31:26.667198 1580 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:26.764178 1733 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:26.616879 4035 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.790498 1728 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260927 23:31:26.762229 4025 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.026s sys 0.018s
I20260927 23:31:26.793468 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:26.620340 1657 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.009s sys 0.035s
W20260927 23:31:26.794890 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 141934us
W20260927 23:31:26.836133 1657 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.146s user 0.009s sys 0.035s
I20260927 23:31:26.656592 3654 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.813948 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:26.803264 1702 leader_election.cc:334] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.362s (SENT)
W20260927 23:31:26.821244 1628 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.004s user 0.014s sys 0.037s
W20260927 23:31:26.828796 1470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 69300.3us
W20260927 23:31:26.838822 1628 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.022s user 0.015s sys 0.037s
W20260927 23:31:26.839187 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 1.0222e+06us
I20260927 23:31:26.834350 1587 raft_consensus.cc:1261] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 23:31:26.864480 1628 leader_election.cc:334] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.871s (SENT)
I20260927 23:31:26.833527 1486 catalog_manager.cc:5495] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:26.865259 1628 leader_election.cc:334] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.309s (SENT)
I20260927 23:31:26.870600 1588 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:26.837231 1583 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:26.832041 1735 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:26.800843 3763 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
W20260927 23:31:26.851013 1654 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.295s user 0.089s sys 0.143s
I20260927 23:31:26.851460 4016 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:26.839941 1734 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:26.815344 4049 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.863821 3825 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.856433 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.856571 4040 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.835203 4080 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.798462 4079 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:26.841635 4025 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s user 0.026s sys 0.018s
W20260927 23:31:26.898012 1732 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.044s user 0.018s sys 0.033s
I20260927 23:31:26.873840 4074 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:26.839569 3875 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:26.877332 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:26.890574 4092 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.879508 1486 catalog_manager.cc:5495] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:26.887193 1628 connection.cc:232] RPC call timeout handler was delayed by -1.0607893709999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20260927 23:31:26.889075 1654 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.333s user 0.089s sys 0.143s
I20260927 23:31:26.886157 1581 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:26.964594 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.889430 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.852126 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.801405 4069 consensus_queue.cc:1046] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260927 23:31:26.830183 3873 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:26.969645 3873 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.857036 3654 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:26.907792 1585 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:26.835325 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:26.974489 1580 raft_consensus.cc:2362] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
W20260927 23:31:26.975630 1628 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.060s (SENT)
I20260927 23:31:26.974483 2290 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:26.976505 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:26.909355 1732 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.055s user 0.018s sys 0.034s
I20260927 23:31:26.878613 4102 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:26.976293 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.793808 3986 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.898702 3825 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.913286 4074 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.915900 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:26.981424 1733 raft_consensus.cc:2379] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:26.916923 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1547) took 3325 ms. Trace:
I20260927 23:31:26.982545 1658 rpcz_store.cc:276] 0927 23:31:23.591481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.667885 (+ 76404us) service_pool.cc:227] Handling call
0927 23:31:23.715817 (+ 47932us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:23.716463 (+ 646us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:23.716504 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:25.468203 (+1751699us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b2c0005b518. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:25.469576 (+ 1373us) log.cc:842] Serialized 52 byte log entry
0927 23:31:26.447636 (+978060us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:26.447690 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:26.447853 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:26.916649 (+468796us) raft_consensus.cc:1671] finished
0927 23:31:26.916668 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:26.916871 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":238,"raft.queue_time_us":692,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":19241,"spinlock_wait_cycles":47480192,"thread_start_us":977652,"threads_started":1,"wal-append.queue_time_us":1410848}
I20260927 23:31:26.918206 3707 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:26.919495 4035 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:26.923260 4050 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.927440 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.927799 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1400) took 1704 ms. Trace:
I20260927 23:31:26.894680 4016 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:26.931555 3732 consensus_meta.cc:287] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.937s user 0.010s sys 0.001s
I20260927 23:31:26.931661 4080 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.909953 4079 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:26.933807 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:26.995282 1661 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:26.936421 3975 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:26.939934 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1557) took 1271 ms. Trace:
I20260927 23:31:26.996156 1662 rpcz_store.cc:276] 0927 23:31:25.668024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.740425 (+ 72401us) service_pool.cc:227] Handling call
0927 23:31:26.939738 (+1199313us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400351e54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:26.939862 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037038976}
W20260927 23:31:26.945050 2289 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 498) took 2323 ms (client timeout 2140 ms). Trace:
W20260927 23:31:26.949508 4111 log.cc:924] Time spent T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.086s user 0.000s sys 0.002s
I20260927 23:31:26.953963 3687 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.966306 1583 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:26.894814 1728 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:26.969336 1486 catalog_manager.cc:5495] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:26.971305 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.013170 1582 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:27.014302 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.015079 1582 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:27.016036 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.016676 1582 raft_consensus.cc:2454] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:27.017531 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.999421 3825 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.023352 2259 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:26.972469 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1744) took 1603 ms. Trace:
I20260927 23:31:27.025269 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:27.025457 1654 rpcz_store.cc:276] 0927 23:31:25.369163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.554597 (+185434us) service_pool.cc:227] Handling call
0927 23:31:25.554987 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:25.555235 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:25.555560 (+ 325us) log.cc:842] Serialized 14 byte log entry
0927 23:31:26.972033 (+1416473us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:26.972105 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:26.972184 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:26.972266 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:26.972400 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1167,"prepare.queue_time_us":4404,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":10222,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":1302,"thread_start_us":1407908,"threads_started":1}
W20260927 23:31:26.978518 2285 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.991s user 0.019s sys 0.015s
I20260927 23:31:26.978057 1581 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:26.994964 4057 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:26.981379 1735 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:26.997913 3732 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:26.986912 1587 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:26.970731 4092 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:26.986878 1737 rpcz_store.cc:276] 0927 23:31:25.223101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.223308 (+ 207us) service_pool.cc:227] Handling call
0927 23:31:25.225706 (+ 2398us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:25.226077 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:25.226120 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:25.255400 (+ 29280us) log.cc:842] Serialized 52 byte log entry
0927 23:31:25.861451 (+606051us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:25.861501 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:25.861586 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:26.927532 (+1065946us) raft_consensus.cc:1671] finished
0927 23:31:26.927556 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:26.927727 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5648,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":1734,"thread_start_us":581257,"threads_started":1,"wal-append.queue_time_us":1347807}
I20260927 23:31:26.999600 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:27.005112 1552 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:46514 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 1 similar messages]
I20260927 23:31:26.977895 4102 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:26.987004 3707 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:26.976570 1628 leader_election.cc:334] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.157s (SENT)
I20260927 23:31:26.863734 3994 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:26.972537 1702 leader_election.cc:334] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.683s (SENT)
I20260927 23:31:27.029874 1486 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:27.042102 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65553.5us
I20260927 23:31:26.976293 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:27.043289 1731 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s user 0.011s sys 0.022s
W20260927 23:31:27.033574 1703 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.017s sys 0.024s
W20260927 23:31:27.043686 1731 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.021s user 0.011s sys 0.022s
W20260927 23:31:27.043788 1703 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.775s user 0.017s sys 0.024s
I20260927 23:31:27.042188 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:27.018195 1582 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:27.038740 2294 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:27.045661 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:27.046676 1582 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:27.026813 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.049654 1588 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:27.050834 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.051863 1588 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:27.052847 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:26.917889 3862 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:27.042922 1628 leader_election.cc:334] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.196s (SENT)
I20260927 23:31:27.053889 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.067941 1581 raft_consensus.cc:2379] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:27.055092 1585 raft_consensus.cc:2396] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:27.069895 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.071226 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.038352 1583 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:27.055364 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:27.073236 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:27.028993 2285 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.042s user 0.020s sys 0.016s
I20260927 23:31:27.062669 1552 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:46514 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1405)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1487)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1412)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1375)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1377)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1413)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 370)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1491)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1494)
W20260927 23:31:27.066859 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 94354.8us
I20260927 23:31:27.079595 1587 raft_consensus.cc:2396] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 23:31:27.034945 3907 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:27.032346 3687 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:27.080890 1585 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:27.081195 1486 catalog_manager.cc:5495] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:27.079630 1702 leader_election.cc:334] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.722s (SENT)
I20260927 23:31:27.092242 1702 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:26.998459 2289 rpcz_store.cc:269] 0927 23:31:24.621473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.624354 (+ 2881us) service_pool.cc:227] Handling call
0927 23:31:26.944788 (+2320434us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540036d3d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:26.944934 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3435500416}
I20260927 23:31:27.035032 3825 consensus_queue.cc:1035] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.054s
I20260927 23:31:27.039404 4048 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:27.096330 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.102869 1581 raft_consensus.cc:2362] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7. Re-sending same reply.
I20260927 23:31:27.068948 3975 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:27.105571 3873 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.087216 3907 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:27.113440 3907 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:27.097507 1702 leader_election.cc:334] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.739s (SENT)
I20260927 23:31:27.128060 1702 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:27.131461 3825 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:27.132046 3825 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:27.132701 4025 raft_consensus.cc:3032] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:27.133213 4025 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:27.136927 2294 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 447) took 1196 ms. Trace:
I20260927 23:31:27.098096 1486 catalog_manager.cc:5495] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce reported cstate change: term changed from 1 to 3, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:27.055539 4066 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.008s sys 0.029s
W20260927 23:31:27.129963 1702 leader_election.cc:334] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.804s (SENT)
I20260927 23:31:27.142606 2294 rpcz_store.cc:276] 0927 23:31:25.940611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.944820 (+ 4209us) service_pool.cc:227] Handling call
0927 23:31:25.945128 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:27.054630 (+1109502us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.136675 (+ 82045us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54004e0cd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:27.136860 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2514519936}
I20260927 23:31:27.134076 3825 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.127317 3873 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:27.173662 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:27.174914 1654 raft_consensus.cc:2379] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:27.165572 1702 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:27.176443 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 46523.2us
I20260927 23:31:27.135389 3758 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:27.136304 4074 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:27.180791 1701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.006s sys 0.034s
W20260927 23:31:27.183517 1701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.007s sys 0.034s
I20260927 23:31:27.185948 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.188452 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.188653 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.190567 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.192101 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.194303 1736 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:27.165423 4066 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.132s user 0.009s sys 0.030s
I20260927 23:31:27.195291 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.168749 3986 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.196422 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.189291 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.216569 1728 raft_consensus.cc:2396] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260927 23:31:27.198513 1729 raft_consensus.cc:2396] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260927 23:31:27.204393 1702 leader_election.cc:334] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.094s (SENT)
I20260927 23:31:27.134158 4025 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.151494 3975 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:27.218995 4113 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260927 23:31:27.219928 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:27.220460 3975 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.220656 1737 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:27.221150 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.221668 3707 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.221962 1735 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 23:31:27.222785 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.223883 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.224617 1735 raft_consensus.cc:2396] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:27.224861 3758 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:27.224926 3986 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:27.225329 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:27.225934 1735 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:27.190472 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.226744 1733 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 7: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:27.227011 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1548) took 3214 ms. Trace:
I20260927 23:31:27.227355 1657 rpcz_store.cc:276] 0927 23:31:24.012329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:24.518985 (+506656us) service_pool.cc:227] Handling call
0927 23:31:24.526220 (+ 7235us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:24.874051 (+347831us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:24.874079 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:25.689576 (+815497us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b2c0005b518. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:25.690037 (+ 461us) log.cc:842] Serialized 26 byte log entry
0927 23:31:26.838304 (+1148267us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:26.838375 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:26.838454 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:27.226722 (+388268us) raft_consensus.cc:1671] finished
0927 23:31:27.226746 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.226956 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1678,"raft.queue_time_us":40160,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":24133,"spinlock_wait_cycles":5692672,"thread_start_us":1495122,"threads_started":2,"wal-append.queue_time_us":1350316}
I20260927 23:31:27.230420 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:27.225785 3986 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:27.218251 2285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 468) took 1286 ms. Trace:
W20260927 23:31:27.232999 1702 leader_election.cc:334] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.233s (SENT)
I20260927 23:31:27.240340 1702 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:27.241150 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:27.243335 4117 log.cc:924] Time spent T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260927 23:31:27.244809 4092 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.220480 4102 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:27.246616 3907 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.240156 2285 rpcz_store.cc:276] 0927 23:31:25.931999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.941871 (+ 9872us) service_pool.cc:227] Handling call
0927 23:31:25.977725 (+ 35854us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:25.977997 (+ 272us) raft_consensus.cc:1520] Early marking committed up to index 8
0927 23:31:25.978300 (+ 303us) log.cc:842] Serialized 14 byte log entry
0927 23:31:27.129675 (+1151375us) log.cc:842] Serialized 14 byte log entry
0927 23:31:27.162288 (+ 32613us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:27.168403 (+ 6115us) raft_consensus.cc:1631] Marking committed up to 8
0927 23:31:27.170694 (+ 2291us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.177819 (+ 7125us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.218175 (+ 40356us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4519,"prepare.queue_time_us":878096,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":11911,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":6589,"thread_start_us":1087865,"threads_started":1}
I20260927 23:31:27.250489 3862 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.179775 3873 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:27.297374 3907 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.222205 3707 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:27.236490 4016 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.293308 3873 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:27.225345 3758 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.213639 4069 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260927 23:31:27.266294 3986 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.310366 4050 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:27.309613 3873 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:27.311646 4025 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:27.224689 4100 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.314600 3707 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:27.314956 3758 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:27.317214 3952 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.022s sys 0.080s
W20260927 23:31:27.324079 3952 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.798s user 0.022s sys 0.081s
I20260927 23:31:27.323961 4100 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:31:27.084735 1552 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46522 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1501)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1375)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1377)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1413)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 370)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1491)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1492)
I20260927 23:31:27.181354 1582 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:27.130904 1588 raft_consensus.cc:2362] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:27.239522 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.263502 1661 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:27.455686 1657 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260927 23:31:27.411379 1551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.909s user 0.006s sys 0.015s
I20260927 23:31:27.525646 2288 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:27.537528 3776 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s user 0.005s sys 0.003s
I20260927 23:31:27.547001 4136 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:27.352056 4029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.005s sys 0.017s
I20260927 23:31:27.468557 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 338) took 1501 ms. Trace:
I20260927 23:31:27.456866 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:27.555305 3986 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:27.555289 2258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.007s sys 0.015s
W20260927 23:31:27.556769 3431 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.816s user 0.009s sys 0.013s
W20260927 23:31:27.350768 4019 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.009s sys 0.014s
I20260927 23:31:27.431711 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:27.456096 1663 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.017s sys 0.014s
I20260927 23:31:27.426122 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:27.583655 1551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.082s user 0.006s sys 0.016s
W20260927 23:31:27.579080 3776 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.006s sys 0.004s
W20260927 23:31:27.592406 1663 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.051s user 0.017s sys 0.014s
W20260927 23:31:27.586583 3922 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.008s sys 0.003s
W20260927 23:31:27.590574 2258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.007s sys 0.015s
W20260927 23:31:27.592388 3431 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.852s user 0.009s sys 0.013s
I20260927 23:31:27.324410 3873 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:27.603753 3922 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.750s user 0.009s sys 0.003s
I20260927 23:31:27.581435 1731 rpcz_store.cc:276] 0927 23:31:25.967099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.967743 (+ 644us) service_pool.cc:227] Handling call
0927 23:31:25.968387 (+ 644us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:25.972763 (+ 4376us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:25.972809 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:25.988560 (+ 15751us) log.cc:842] Serialized 26 byte log entry
0927 23:31:27.044210 (+1055650us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:27.044262 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.044348 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:27.467883 (+423535us) raft_consensus.cc:1671] finished
0927 23:31:27.467943 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.468271 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":153,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":5254,"thread_start_us":1021776,"threads_started":1,"wal-append.queue_time_us":1326444}
W20260927 23:31:27.619352 4029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.848s user 0.006s sys 0.018s
I20260927 23:31:27.622663 1588 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
W20260927 23:31:27.622409 3900 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.012s sys 0.019s
W20260927 23:31:27.632570 1656 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.000s sys 0.023s
W20260927 23:31:27.641863 1627 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.002s sys 0.029s
W20260927 23:31:27.639182 4019 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.010s sys 0.015s
W20260927 23:31:27.642270 1627 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.002s sys 0.029s
W20260927 23:31:27.642773 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 843447us
W20260927 23:31:27.637336 1730 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.001s sys 0.009s
W20260927 23:31:27.633728 1700 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.011s sys 0.014s
W20260927 23:31:27.652140 1656 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.000s sys 0.024s
W20260927 23:31:27.641944 1554 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.011s sys 0.033s
W20260927 23:31:27.658728 1700 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.012s sys 0.014s
I20260927 23:31:27.659320 4130 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:27.660396 4130 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.637949 4142 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:27.666935 4142 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.668512 4130 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:27.663187 1732 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.000s sys 0.036s
W20260927 23:31:27.665359 1554 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.778s user 0.012s sys 0.033s
W20260927 23:31:27.670501 1730 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.002s sys 0.010s
W20260927 23:31:27.651870 1655 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s user 0.001s sys 0.009s
W20260927 23:31:27.664824 1703 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.009s sys 0.021s
W20260927 23:31:27.672887 1660 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.004s sys 0.014s
W20260927 23:31:27.670874 1584 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.009s sys 0.020s
W20260927 23:31:27.657310 3900 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s user 0.013s sys 0.019s
W20260927 23:31:27.657233 1734 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.015s sys 0.022s
I20260927 23:31:27.684466 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:27.686074 1703 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.619s user 0.009s sys 0.023s
I20260927 23:31:27.685297 2291 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:27.690166 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:27.685072 2289 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:27.688668 1584 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.632s user 0.010s sys 0.021s
W20260927 23:31:27.693377 1732 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.000s sys 0.038s
W20260927 23:31:27.687914 1660 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.730s user 0.005s sys 0.014s
I20260927 23:31:27.624810 4136 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:27.693751 1655 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s user 0.002s sys 0.010s
W20260927 23:31:27.682186 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 760076us
I20260927 23:31:27.702198 2294 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:27.702469 3994 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:27.703238 3994 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.706086 1737 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
W20260927 23:31:27.706506 1702 leader_election.cc:334] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.893s (SENT)
W20260927 23:31:27.697793 1734 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s user 0.016s sys 0.022s
W20260927 23:31:27.710876 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 633778us
I20260927 23:31:27.706667 4142 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:27.721395 4040 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:27.722246 4040 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.723045 1731 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260927 23:31:27.719717 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 833105us
I20260927 23:31:27.723662 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.710350 3994 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:27.726982 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 108545us
I20260927 23:31:27.730994 3873 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:27.740978 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:27.751065 4126 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:27.753500 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:27.740869 3873 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.755594 1735 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:27.764716 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.765020 2294 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:27.763646 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:27.766443 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:27.766727 1702 leader_election.cc:334] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.552s (SENT)
I20260927 23:31:27.767382 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:27.767781 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.768112 3986 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:27.768546 4040 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.769021 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:27.769223 3986 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:27.776233 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 46314us
I20260927 23:31:27.761824 3707 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:27.761807 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:27.773051 1661 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:27.782193 3707 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:27.782297 4126 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:27.784909 4126 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.784970 3707 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:27.782712 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:27.785248 1628 leader_election.cc:334] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.802s (SENT)
I20260927 23:31:27.785753 2289 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:27.786966 3873 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:27.779985 1658 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:27.417254 1553 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46522 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1501)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1375)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1377)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1413)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 370)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1490)
I20260927 23:31:27.781237 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1396) took 2616 ms. Trace:
W20260927 23:31:27.801283 1702 leader_election.cc:334] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.212s (SENT)
W20260927 23:31:27.791416 4151 log.cc:924] Time spent T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260927 23:31:27.821817 1702 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:27.819108 4136 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.837244 1627 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:27.847774 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 102501us
I20260927 23:31:27.851120 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:27.859318 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 60537.9us
I20260927 23:31:27.863255 1737 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:27.795083 1732 rpcz_store.cc:276] 0927 23:31:25.162521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:25.162670 (+ 149us) service_pool.cc:227] Handling call
0927 23:31:25.171687 (+ 9017us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:25.171904 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:25.853951 (+682047us) log.cc:842] Serialized 14 byte log entry
0927 23:31:27.730254 (+1876303us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:27.742071 (+ 11817us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:27.755037 (+ 12966us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.762528 (+ 7491us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.779186 (+ 16658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18533,"prepare.queue_time_us":62478,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":134712,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":82,"thread_start_us":2517776,"threads_started":3,"wal-append.queue_time_us":1508556}
I20260927 23:31:27.871940 4125 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:27.887507 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.888652 1585 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:27.890947 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:27.892592 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:27.895072 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 101571us
I20260927 23:31:27.896726 4125 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:27.897248 4125 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:27.897588 4125 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:27.898208 4149 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.133s user 0.000s sys 0.003s
I20260927 23:31:27.900300 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1373) took 1457 ms. Trace:
I20260927 23:31:27.889981 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.810335 1729 raft_consensus.cc:2932] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:27.885149 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 100042us
W20260927 23:31:27.907835 4141 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
W20260927 23:31:27.908191 4025 consensus_meta.cc:287] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.596s user 0.003s sys 0.010s
I20260927 23:31:27.909250 1657 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 23:31:27.910038 4154 log.cc:924] Time spent T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260927 23:31:27.912324 4019 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:27.913131 4019 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:27.925164 4136 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 23:31:27.927040 2290 consensus_meta.cc:287] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.949s user 0.014s sys 0.006s
I20260927 23:31:27.929493 4019 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:27.932020 4050 consensus_meta.cc:287] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.621s user 0.013s sys 0.002s
I20260927 23:31:27.900753 1583 rpcz_store.cc:276] 0927 23:31:26.443067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:27.074300 (+631233us) service_pool.cc:227] Handling call
0927 23:31:27.100289 (+ 25989us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:27.100707 (+ 418us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:27.100749 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:27.129949 (+ 29200us) log.cc:842] Serialized 26 byte log entry
0927 23:31:27.682823 (+552874us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:27.694179 (+ 11356us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.707704 (+ 13525us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:27.899825 (+192121us) raft_consensus.cc:1671] finished
0927 23:31:27.899845 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.900087 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1625,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":1555,"thread_start_us":436692,"threads_started":1,"wal-append.queue_time_us":631763}
I20260927 23:31:27.904942 4137 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:27.935722 3687 consensus_meta.cc:287] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.840s user 0.014s sys 0.003s
I20260927 23:31:27.916659 2288 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 505) took 1238 ms. Trace:
W20260927 23:31:27.943686 1628 leader_election.cc:334] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.020s (SENT)
W20260927 23:31:27.951366 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 54872.4us
I20260927 23:31:27.954289 1585 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
W20260927 23:31:27.966733 1702 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.255s (SENT)
I20260927 23:31:27.958117 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1415) took 1048 ms. Trace:
I20260927 23:31:27.965392 3763 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:27.973497 1586 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:27.973698 1736 raft_consensus.cc:1203] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[5.3-5.3] Dedup: 5.3->[]
I20260927 23:31:27.974291 1736 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 23:31:27.974032 3763 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:27.974728 3758 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:27.976402 3763 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:27.977101 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:27.949720 2288 rpcz_store.cc:276] 0927 23:31:26.666939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.683404 (+ 16465us) service_pool.cc:227] Handling call
0927 23:31:26.715752 (+ 32348us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:27.592727 (+876975us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.900691 (+307964us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54006437d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:27.902423 (+ 1732us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1747243392}
I20260927 23:31:27.959597 4136 raft_consensus.cc:686] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:27.973134 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:27.985308 4127 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.246s user 0.002s sys 0.000s
I20260927 23:31:27.986785 3758 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:27.987502 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 267) took 1447 ms. Trace:
I20260927 23:31:27.987982 1663 rpcz_store.cc:276] 0927 23:31:26.539800 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.540162 (+ 362us) service_pool.cc:227] Handling call
0927 23:31:26.540500 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:26.540698 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:26.540732 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:26.541150 (+ 418us) log.cc:842] Serialized 26 byte log entry
0927 23:31:27.639690 (+1098540us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:27.659349 (+ 19659us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.659496 (+ 147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:27.984485 (+324989us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:27.987054 (+ 2569us) raft_consensus.cc:1671] finished
0927 23:31:27.987078 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.987415 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3238272,"thread_start_us":1056193,"threads_started":1,"wal-append.queue_time_us":1132672}
W20260927 23:31:27.990882 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 54570.8us
I20260927 23:31:27.972200 4137 raft_consensus.cc:686] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:27.950016 3922 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20260927 23:31:27.995800 1583 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:28.004070 3825 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:28.011619 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 192460us
I20260927 23:31:28.012120 2290 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 506) took 1299 ms. Trace:
I20260927 23:31:28.012607 2290 rpcz_store.cc:276] 0927 23:31:26.712512 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.941479 (+228967us) service_pool.cc:227] Handling call
0927 23:31:26.968049 (+ 26570us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.010628 (+1042579us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.010693 (+ 65us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:28.010753 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:28.010803 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.010897 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:28.011899 (+ 1002us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b540036d3d0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.012058 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72960,"raft.queue_time_us":82790,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"spinlock_wait_cycles":52423936}
I20260927 23:31:28.013919 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.014549 4035 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:28.014751 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.016271 3431 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:28.016136 3763 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:28.016834 3431 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:28.017323 3431 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:28.017318 3763 raft_consensus.cc:3032] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:28.019551 3763 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:28.019785 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 53121.8us
I20260927 23:31:28.020592 3763 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.022014 3763 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:28.022737 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.024288 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:28.029282 1628 leader_election.cc:334] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.405s (SENT)
W20260927 23:31:28.032968 4079 consensus_meta.cc:287] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.023s user 0.009s sys 0.006s
I20260927 23:31:28.033469 4079 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.034888 1580 raft_consensus.cc:2379] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:28.038628 4079 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:27.969113 1734 rpcz_store.cc:276] 0927 23:31:26.910135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.910991 (+ 856us) service_pool.cc:227] Handling call
0927 23:31:26.912777 (+ 1786us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:26.913146 (+ 369us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:26.913191 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:26.913748 (+ 557us) log.cc:842] Serialized 52 byte log entry
0927 23:31:27.749559 (+835811us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:27.940708 (+191149us) log.cc:842] Serialized 14 byte log entry
0927 23:31:27.945910 (+ 5202us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.957348 (+ 11438us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:27.957415 (+ 67us) raft_consensus.cc:1671] finished
0927 23:31:27.957432 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:27.958046 (+ 614us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":206754,"raft.queue_time_us":10184,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":16126,"thread_start_us":939841,"threads_started":2,"wal-append.queue_time_us":839954}
I20260927 23:31:28.042644 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.044174 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.044967 1583 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:28.049832 4069 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 23:31:28.050725 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.052027 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.052614 1581 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:28.053112 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.054912 2259 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:28.057755 4152 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.189s user 0.000s sys 0.002s
I20260927 23:31:28.058739 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:28.058903 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.059056 4137 consensus_queue.cc:237] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:28.061272 2287 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 425) took 4662 ms (client timeout 2098 ms). Trace:
I20260927 23:31:28.061033 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:28.061833 2287 rpcz_store.cc:269] 0927 23:31:23.398434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.403029 (+ 4595us) service_pool.cc:227] Handling call
0927 23:31:28.060662 (+4657633us) spinlock_profiling.cc:243] Waited 3.93 s on lock 0x7b540069e6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.061063 (+ 401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9027661568}
I20260927 23:31:28.063829 4130 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:28.064380 4130 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:28.065270 4145 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.197s user 0.000s sys 0.001s
I20260927 23:31:28.066356 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.066776 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.067538 4130 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:28.067849 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:28.068646 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 270) took 1191 ms. Trace:
I20260927 23:31:28.069202 1656 rpcz_store.cc:276] 0927 23:31:26.877194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.877363 (+ 169us) service_pool.cc:227] Handling call
0927 23:31:26.887522 (+ 10159us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:26.887923 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:26.887960 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:26.893374 (+ 5414us) log.cc:842] Serialized 26 byte log entry
0927 23:31:27.716643 (+823269us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:27.726441 (+ 9798us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:27.726517 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:28.064498 (+337981us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:28.068289 (+ 3791us) raft_consensus.cc:1671] finished
0927 23:31:28.068316 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:28.068575 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23947,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":25736576,"thread_start_us":684573,"threads_started":1,"wal-append.queue_time_us":892829}
I20260927 23:31:28.068688 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:28.071923 4048 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.967s user 0.018s sys 0.000s
I20260927 23:31:28.073438 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.073724 4136 consensus_queue.cc:237] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:28.078646 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1b945905616b4989853b0a92bfbd6b98 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:28.081751 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:28.088263 1579 raft_consensus.cc:2354] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 8.
I20260927 23:31:28.087554 4125 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.089326 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:28.092666 4157 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.329s user 0.000s sys 0.001s
I20260927 23:31:28.093664 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1371) took 1653 ms. Trace:
I20260927 23:31:28.103024 4050 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.104043 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1397) took 1571 ms. Trace:
I20260927 23:31:28.103739 4050 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.104517 1580 rpcz_store.cc:276] 0927 23:31:26.532350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.975990 (+443640us) service_pool.cc:227] Handling call
0927 23:31:28.103763 (+1127773us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400242554. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.103955 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432064512}
I20260927 23:31:28.107455 4050 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:28.108700 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.102654 1584 rpcz_store.cc:276] 0927 23:31:26.439846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.440461 (+ 615us) service_pool.cc:227] Handling call
0927 23:31:26.497489 (+ 57028us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:26.520277 (+ 22788us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:26.520320 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:26.549328 (+ 29008us) log.cc:842] Serialized 26 byte log entry
0927 23:31:27.031659 (+482331us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:27.737306 (+705647us) log.cc:842] Serialized 14 byte log entry
0927 23:31:27.763907 (+ 26601us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.093149 (+329242us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54003b42d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:28.093198 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:28.093268 (+ 70us) raft_consensus.cc:1671] finished
0927 23:31:28.093387 (+ 119us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:28.093581 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51077,"prepare.queue_time_us":26168,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":39488,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":43607,"spinlock_wait_cycles":51454208,"thread_start_us":1077745,"threads_started":2,"wal-append.queue_time_us":757219}
I20260927 23:31:28.110751 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.111433 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.111797 1584 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:28.113976 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:28.115590 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1384) took 1669 ms (client timeout 1871 ms). Trace:
W20260927 23:31:28.116007 1586 rpcz_store.cc:269] 0927 23:31:26.446171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.657020 (+210849us) service_pool.cc:227] Handling call
0927 23:31:28.115411 (+1458391us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540013fbd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.115542 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2756577536}
I20260927 23:31:28.116452 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.117000 1585 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.117204 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.117539 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1512) took 1033 ms. Trace:
I20260927 23:31:28.117779 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1501) took 1042 ms. Trace:
I20260927 23:31:28.117969 1586 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 7: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:28.119113 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1502) took 1043 ms. Trace:
I20260927 23:31:28.119485 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1368) took 1681 ms. Trace:
I20260927 23:31:28.119411 1586 rpcz_store.cc:276] 0927 23:31:27.076092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.116864 (+1040772us) service_pool.cc:227] Handling call
0927 23:31:28.119063 (+ 2199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.119876 1579 rpcz_store.cc:276] 0927 23:31:26.437826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:26.437945 (+ 119us) service_pool.cc:227] Handling call
0927 23:31:28.119305 (+1681360us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540009f6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.119435 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3505990400}
I20260927 23:31:28.120334 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.120823 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.121106 1586 raft_consensus.cc:2396] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 23:31:28.121647 1579 raft_consensus.cc:2396] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 23:31:28.097822 1587 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.122551 3761 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.122686 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1510) took 1039 ms. Trace:
I20260927 23:31:28.122999 1587 rpcz_store.cc:276] 0927 23:31:27.083064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:27.898214 (+815150us) service_pool.cc:227] Handling call
0927 23:31:28.122625 (+224411us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.123026 3761 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.121922 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.118402 1584 rpcz_store.cc:276] 0927 23:31:27.075816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.116170 (+1040354us) service_pool.cc:227] Handling call
0927 23:31:28.117701 (+ 1531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:27.417147 1552 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:46522 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 374)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1515)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1501)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1374)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1375)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1377)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1379)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1390)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1391)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1392)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1393)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1413)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 360)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 361)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 362)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 364)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 365)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 366)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 370)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 371)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1490)
I20260927 23:31:28.045962 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.093015 3431 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.130841 3761 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:28.130044 4057 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.061007 4040 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:28.117904 1585 rpcz_store.cc:276] 0927 23:31:27.083731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:27.969433 (+885702us) service_pool.cc:227] Handling call
0927 23:31:28.117491 (+148058us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.106137 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.147547 1580 raft_consensus.cc:2379] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 3: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:28.147778 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1505) took 1070 ms. Trace:
I20260927 23:31:28.148039 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1499) took 1072 ms. Trace:
I20260927 23:31:28.148319 1586 rpcz_store.cc:276] 0927 23:31:27.076957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.121667 (+1044710us) service_pool.cc:227] Handling call
0927 23:31:28.147710 (+ 26043us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.148587 1580 rpcz_store.cc:276] 0927 23:31:27.075184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.105758 (+1030574us) service_pool.cc:227] Handling call
0927 23:31:28.147974 (+ 42216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.149951 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:28.151083 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 125484us
I20260927 23:31:28.153887 1729 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:28.155354 1582 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:28.155036 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:28.159689 1580 raft_consensus.cc:2396] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:28.163307 1583 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:28.163847 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1507) took 1086 ms. Trace:
I20260927 23:31:28.164001 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.164209 1583 rpcz_store.cc:276] 0927 23:31:27.077517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.045634 (+968117us) service_pool.cc:227] Handling call
0927 23:31:28.163791 (+118157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.164767 1588 raft_consensus.cc:2454] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:28.165228 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.165953 1588 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 2. Current term is 4. Ops: []
I20260927 23:31:28.167426 3431 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:28.168288 1663 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260927 23:31:28.169759 4161 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20260927 23:31:28.172705 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.173578 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.160347 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.174417 1585 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 2.
I20260927 23:31:28.165979 1583 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:28.128545 1579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "879c1897331c4c32ade3f828689195c4"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:28.177754 1579 raft_consensus.cc:595] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:28.171286 1628 leader_election.cc:334] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.769s (SENT)
I20260927 23:31:28.180367 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1376) took 1728 ms. Trace:
I20260927 23:31:28.180848 1588 rpcz_store.cc:276] 0927 23:31:26.443916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.165386 (+1721470us) service_pool.cc:227] Handling call
0927 23:31:28.165768 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.172182 (+ 6414us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.172397 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.178239 1579 raft_consensus.cc:603] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:28.186203 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:28.191152 1736 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:28.193707 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.194777 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.196100 1657 raft_consensus.cc:2396] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260927 23:31:28.197216 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.198506 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.198926 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.199425 2289 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 23:31:28.199564 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.202508 1736 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 23:31:28.203249 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.204008 1731 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:28.206952 1584 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:28.207746 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.199057 1580 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:28.215875 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1377) took 1764 ms. Trace:
I20260927 23:31:28.216210 1580 rpcz_store.cc:276] 0927 23:31:26.444214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.172136 (+1727922us) service_pool.cc:227] Handling call
0927 23:31:28.198727 (+ 26591us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.208446 (+ 9719us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.208617 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.196097 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.174263 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.218823 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1375) took 1772 ms. Trace:
I20260927 23:31:28.219247 1584 rpcz_store.cc:276] 0927 23:31:26.443645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.131962 (+1688317us) service_pool.cc:227] Handling call
0927 23:31:28.182122 (+ 50160us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.215419 (+ 33297us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.215542 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.221079 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:28.222940 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.223565 1584 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.150830 4057 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:28.232949 1581 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:28.237751 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.238452 1580 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.241541 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.242897 1584 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
W20260927 23:31:28.264761 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 107314us
I20260927 23:31:28.227545 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:28.248210 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:28.294824 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:28.414794 1629 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:28.395252 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 145665us
I20260927 23:31:28.303268 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.433275 1584 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:28.297315 1582 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:28.434345 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56820 (request call id 374) took 1018 ms. Trace:
I20260927 23:31:28.436959 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.442924 1585 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260927 23:31:28.444671 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 268807us
I20260927 23:31:28.306713 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.427510 1554 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:28.434930 1582 rpcz_store.cc:276] 0927 23:31:27.416419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.081383 (+664964us) service_pool.cc:227] Handling call
0927 23:31:28.434272 (+352889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:28.424081 4125 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:28.287449 4130 consensus_queue.cc:1046] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.174s
I20260927 23:31:28.307307 3994 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.169s
I20260927 23:31:28.572911 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1379) took 2113 ms. Trace:
I20260927 23:31:28.679476 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:28.539999 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 69945.5us
I20260927 23:31:28.576485 1729 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:56992
I20260927 23:31:28.446142 2289 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
W20260927 23:31:28.527496 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 279197us
I20260927 23:31:28.489684 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:28.477895 1580 raft_consensus.cc:2454] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.576710 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1391) took 2067 ms. Trace:
W20260927 23:31:28.514691 1702 leader_election.cc:334] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.505s (SENT)
I20260927 23:31:28.687141 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.690163 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.694669 1656 raft_consensus.cc:2396] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 23:31:28.695786 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.695700 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.694649 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.702818 1582 raft_consensus.cc:2396] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:28.683686 1585 rpcz_store.cc:276] 0927 23:31:26.444744 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.183129 (+1738385us) service_pool.cc:227] Handling call
0927 23:31:28.436250 (+253121us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.450055 (+ 13805us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.558374 (+108319us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400330ed0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:28.558541 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457418496}
I20260927 23:31:28.450910 4136 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260927 23:31:28.575191 4174 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260927 23:31:28.686818 1729 raft_consensus.cc:595] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:28.574905 3952 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.488857 3761 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:28.716491 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.757925 1581 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
W20260927 23:31:28.716576 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 278895us
I20260927 23:31:28.717299 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:28.573470 3776 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.688953 1588 rpcz_store.cc:276] 0927 23:31:26.513741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.183445 (+1669704us) service_pool.cc:227] Handling call
0927 23:31:28.185340 (+ 1895us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.212490 (+ 27150us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.212533 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.219422 (+ 6889us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.398928 (+179506us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:28.398979 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.399054 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:28.576480 (+177426us) raft_consensus.cc:1671] finished
0927 23:31:28.576502 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:28.576659 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":120393,"threads_started":1,"wal-append.queue_time_us":313824}
I20260927 23:31:28.544559 3431 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:28.574538 3763 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:28.754101 1584 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:28.628114 4079 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:28.754045 1729 raft_consensus.cc:603] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:28.758901 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:28.762113 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:28.775542 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 314348us
I20260927 23:31:28.556815 4057 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:28.786983 1554 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:28.789564 1580 raft_consensus.cc:2379] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:28.724853 4136 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:28.490981 4048 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.297s
W20260927 23:31:28.843236 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 128939us
I20260927 23:31:28.854205 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:28.870517 1581 raft_consensus.cc:1261] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 23:31:28.827895 3431 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:28.868081 2259 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:28.787460 3952 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:29.237659 1728 consensus_meta.cc:287] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.944s user 0.008s sys 0.001s
W20260927 23:31:28.936272 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 57343.1us
I20260927 23:31:29.238189 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:29.008383 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 142166us
I20260927 23:31:28.846805 4136 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:29.233055 1733 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:29.248126 1728 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:29.249174 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1398) took 2716 ms. Trace:
I20260927 23:31:29.250926 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:28.895820 1584 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 23:31:29.233196 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 718487us
I20260927 23:31:28.869982 3761 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:28.847041 4057 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:29.273723 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1408) took 2738 ms. Trace:
I20260927 23:31:29.283396 1584 rpcz_store.cc:276] 0927 23:31:26.535769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.791191 (+2255422us) service_pool.cc:227] Handling call
0927 23:31:28.895426 (+104235us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.273421 (+377995us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.273662 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.272218 1581 rpcz_store.cc:276] 0927 23:31:26.532626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.758623 (+2225997us) service_pool.cc:227] Handling call
0927 23:31:28.867059 (+108436us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.870945 (+ 3886us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.249097 (+378152us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:29.226105 4177 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.381s user 0.000s sys 0.000s
W20260927 23:31:29.225160 4176 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.432s user 0.000s sys 0.001s
I20260927 23:31:29.233896 4195 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:28.853950 3763 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:29.352415 2258 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:29.181058 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1399) took 2648 ms. Trace:
W20260927 23:31:29.079650 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 225531us
W20260927 23:31:29.359866 1734 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.007s sys 0.001s
I20260927 23:31:29.365294 1588 rpcz_store.cc:276] 0927 23:31:26.532891 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.787606 (+2254715us) service_pool.cc:227] Handling call
0927 23:31:28.895561 (+107955us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.180535 (+284974us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:29.180582 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:29.180660 (+ 78us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:29.180711 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.180809 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.180981 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:29.367245 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 48842.1us
W20260927 23:31:29.367623 1734 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.008s sys 0.001s
I20260927 23:31:29.318140 4125 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:29.324915 3707 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:28.871253 3922 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.479s
I20260927 23:31:28.916834 4079 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:29.406922 1581 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:29.407774 1584 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260927 23:31:29.100725 4141 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20260927 23:31:29.395298 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 97635.2us
I20260927 23:31:29.396782 1582 raft_consensus.cc:2396] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 23:31:29.345935 4057 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:29.345793 4195 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:29.398491 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 551919us
I20260927 23:31:28.916867 3776 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:29.155241 4069 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:29.419173 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:29.499732 1656 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:29.412189 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1414) took 2874 ms. Trace:
I20260927 23:31:29.412667 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1418) took 2874 ms. Trace:
I20260927 23:31:29.500727 1584 rpcz_store.cc:276] 0927 23:31:26.537253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.307745 (+2770492us) service_pool.cc:227] Handling call
0927 23:31:29.385824 (+ 78079us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.408220 (+ 22396us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.412109 (+ 3889us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.500873 1581 rpcz_store.cc:276] 0927 23:31:26.538289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.307714 (+2769425us) service_pool.cc:227] Handling call
0927 23:31:29.385485 (+ 77771us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.407414 (+ 21929us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.412606 (+ 5192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.423454 1703 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:29.073534 4048 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:29.410187 1536 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45868.4us
I20260927 23:31:29.513083 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60802 (request call id 351) took 1318 ms. Trace:
W20260927 23:31:29.450603 1702 leader_election.cc:334] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.572s (SENT)
I20260927 23:31:29.514232 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:29.529899 1629 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:29.513548 1728 rpcz_store.cc:276] 0927 23:31:28.132124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.144550 (+ 12426us) service_pool.cc:227] Handling call
0927 23:31:29.449990 (+1305440us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b54003c3f54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:29.450122 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183894784}
W20260927 23:31:29.537509 1585 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.067s sys 0.106s
W20260927 23:31:29.538717 1585 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.068s sys 0.106s
I20260927 23:31:29.417989 3761 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:29.011307 4137 consensus_queue.cc:1046] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.486s
I20260927 23:31:29.453536 4195 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
W20260927 23:31:29.569466 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 174680us
I20260927 23:31:29.558748 4201 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:29.410187 3707 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:29.551635 4177 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260927 23:31:29.397549 4125 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:29.453104 4057 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:29.605851 4175 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.632s user 0.001s sys 0.000s
W20260927 23:31:29.613019 4187 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260927 23:31:29.594199 4130 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:29.610651 4205 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:29.605270 4183 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:29.611452 4188 consensus_queue.cc:1035] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.304s
I20260927 23:31:29.606770 4190 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.257s
I20260927 23:31:29.562212 3761 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:29.758666 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:29.744957 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 347) took 1618 ms. Trace:
I20260927 23:31:29.712913 1584 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:29.692541 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:29.750244 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1374) took 3306 ms. Trace:
I20260927 23:31:29.560953 4048 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:29.760555 1732 rpcz_store.cc:276] 0927 23:31:28.126384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.126546 (+ 162us) service_pool.cc:227] Handling call
0927 23:31:28.129718 (+ 3172us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.130227 (+ 509us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.130257 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.130791 (+ 534us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.264355 (+133564us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:28.264391 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.264468 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:29.741602 (+1477134us) raft_consensus.cc:1671] finished
0927 23:31:29.744721 (+ 3119us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.744891 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2003,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":14696,"thread_start_us":133217,"threads_started":1,"wal-append.queue_time_us":501329}
W20260927 23:31:29.683509 3431 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.045s sys 0.069s
I20260927 23:31:29.768323 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 360) took 3202 ms. Trace:
W20260927 23:31:29.773741 1737 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.003s sys 0.020s
W20260927 23:31:29.776504 1737 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.898s user 0.003s sys 0.021s
I20260927 23:31:29.776278 1584 rpcz_store.cc:276] 0927 23:31:26.566256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.527270 (+2961014us) service_pool.cc:227] Handling call
0927 23:31:29.672017 (+144747us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.767985 (+ 95968us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.768258 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.780318 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 361) took 3213 ms. Trace:
I20260927 23:31:29.683867 4201 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:29.780845 1581 rpcz_store.cc:276] 0927 23:31:26.566989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.527811 (+2960822us) service_pool.cc:227] Handling call
0927 23:31:29.745357 (+217546us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.779861 (+ 34504us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:29.779898 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:29.779941 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:29.779984 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.780070 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.780249 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.762224 1587 rpcz_store.cc:276] 0927 23:31:26.443368 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.123745 (+1680377us) service_pool.cc:227] Handling call
0927 23:31:28.124183 (+ 438us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.124708 (+ 525us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.124738 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.125207 (+ 469us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.251289 (+126082us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:28.390265 (+138976us) log.cc:842] Serialized 14 byte log entry
0927 23:31:28.393780 (+ 3515us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.393907 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:29.750011 (+1356104us) raft_consensus.cc:1671] finished
0927 23:31:29.750032 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.750178 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":139070,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":1316,"raft.queue_time_us":211,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":5574,"thread_start_us":125684,"threads_started":1,"wal-append.queue_time_us":554715}
W20260927 23:31:29.756379 4199 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20260927 23:31:29.789759 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1574) took 1108 ms. Trace:
I20260927 23:31:29.790948 1655 rpcz_store.cc:276] 0927 23:31:28.673088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.674838 (+ 1750us) service_pool.cc:227] Handling call
0927 23:31:29.608719 (+933881us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:29.781417 (+172698us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3062656}
I20260927 23:31:29.797501 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:29.594345 4069 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:29.776176 3431 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.048s sys 0.072s
I20260927 23:31:29.814181 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:29.824908 1584 raft_consensus.cc:595] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:29.825348 1584 raft_consensus.cc:603] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 23:31:29.827031 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 297250us
I20260927 23:31:29.711592 4130 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:29.841120 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:29.807102 3431 raft_consensus.cc:686] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:29.837121 4191 log.cc:924] Time spent T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
W20260927 23:31:29.858187 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 434564us
I20260927 23:31:29.852768 1661 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:29.870540 1581 raft_consensus.cc:1261] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:29.760141 4205 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:29.878871 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 362) took 3310 ms. Trace:
I20260927 23:31:29.823173 3763 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:29.887295 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 364) took 3276 ms. Trace:
I20260927 23:31:29.902640 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:29.889169 1581 rpcz_store.cc:276] 0927 23:31:26.567783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.806921 (+3239138us) service_pool.cc:227] Handling call
0927 23:31:29.870045 (+ 63124us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.878340 (+ 8295us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.878612 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.890067 1587 rpcz_store.cc:276] 0927 23:31:26.610936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.807463 (+3196527us) service_pool.cc:227] Handling call
0927 23:31:29.853962 (+ 46499us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.875003 (+ 21041us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:29.875054 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:29.886977 (+ 11923us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:29.887045 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.887107 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.887230 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 23:31:29.918602 1735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.006s sys 0.068s
W20260927 23:31:29.937937 1735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.006s sys 0.068s
I20260927 23:31:29.853355 3707 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:29.843469 3952 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:29.941640 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1437) took 1481 ms. Trace:
I20260927 23:31:29.940429 1584 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 7. Ops: []
W20260927 23:31:29.935186 4192 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260927 23:31:29.883635 4079 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:29.947148 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 365) took 3325 ms. Trace:
I20260927 23:31:29.951373 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:29.972494 1584 rpcz_store.cc:276] 0927 23:31:26.621741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.826946 (+3205205us) service_pool.cc:227] Handling call
0927 23:31:29.885838 (+ 58892us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:29.946812 (+ 60974us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.947092 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.975703 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1572) took 1303 ms. Trace:
I20260927 23:31:29.978433 1587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:29.979043 1655 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:29.979195 1587 raft_consensus.cc:595] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:29.979538 1587 raft_consensus.cc:603] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:29.935737 3761 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:29.976130 1660 rpcz_store.cc:276] 0927 23:31:28.672473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.675121 (+ 2648us) service_pool.cc:227] Handling call
0927 23:31:28.694707 (+ 19586us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.786884 (+ 92177us) raft_consensus.cc:1520] Early marking committed up to index 1
0927 23:31:28.786911 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.854860 (+ 67949us) log.cc:842] Serialized 26 byte log entry
0927 23:31:29.363431 (+508571us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:29.363465 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.363549 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:29.975468 (+611919us) raft_consensus.cc:1671] finished
0927 23:31:29.975487 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.975645 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1343,"raft.queue_time_us":22229,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":63458,"thread_start_us":411680,"threads_started":1,"wal-append.queue_time_us":505006}
I20260927 23:31:29.938354 4206 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:29.997182 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:29.883128 4035 consensus_queue.cc:1046] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.102s
I20260927 23:31:29.984352 3707 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:29.999004 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1389) took 3490 ms. Trace:
I20260927 23:31:29.946614 3952 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:29.854306 4208 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.146s
I20260927 23:31:30.015550 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 366) took 3361 ms. Trace:
I20260927 23:31:30.016168 4214 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:30.016848 4214 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:30.018596 4171 log.cc:924] Time spent T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260927 23:31:30.020514 4048 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:30.021381 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 23:31:30.022109 4213 consensus_queue.cc:1035] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 23:31:29.997392 3776 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.016009 1581 rpcz_store.cc:276] 0927 23:31:26.653728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.945824 (+3292096us) service_pool.cc:227] Handling call
0927 23:31:30.004111 (+ 58287us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.015175 (+ 11064us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.015208 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.015271 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:30.015309 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.015375 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.015496 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6030,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115}
I20260927 23:31:30.025586 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 368) took 3307 ms. Trace:
I20260927 23:31:29.972414 3763 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:30.026000 1587 rpcz_store.cc:276] 0927 23:31:26.717766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.000016 (+3282250us) service_pool.cc:227] Handling call
0927 23:31:30.024949 (+ 24933us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.025163 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.025191 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.025253 (+ 62us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:30.025329 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.025398 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.025508 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:30.026820 3763 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.027484 1587 raft_consensus.cc:1226] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:30.028059 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 367) took 3344 ms. Trace:
I20260927 23:31:30.004513 1732 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
W20260927 23:31:30.006851 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 248276us
I20260927 23:31:30.028502 1584 rpcz_store.cc:276] 0927 23:31:26.683674 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.974186 (+3290512us) service_pool.cc:227] Handling call
0927 23:31:30.010174 (+ 35988us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.027749 (+ 17575us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.027775 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.027813 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:30.027852 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.027917 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.028010 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:30.024220 4205 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.024262 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:30.027259 4214 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.006150 1583 rpcz_store.cc:276] 0927 23:31:26.508881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.182372 (+1673491us) service_pool.cc:227] Handling call
0927 23:31:28.233799 (+ 51427us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.246455 (+ 12656us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.246495 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.423766 (+177271us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.661310 (+237544us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:28.701192 (+ 39882us) log.cc:842] Serialized 14 byte log entry
0927 23:31:28.701626 (+ 434us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.701724 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:29.998729 (+1297005us) raft_consensus.cc:1671] finished
0927 23:31:29.998751 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.998909 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":41881,"prepare.run_cpu_time_us":9964,"prepare.run_wall_time_us":37578,"raft.queue_time_us":9202,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":2507,"spinlock_wait_cycles":33208960,"thread_start_us":237107,"threads_started":1,"wal-append.queue_time_us":756087}
I20260927 23:31:30.028120 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 370) took 3156 ms. Trace:
I20260927 23:31:30.031957 1587 rpcz_store.cc:276] 0927 23:31:26.872131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.026978 (+3154847us) service_pool.cc:227] Handling call
0927 23:31:30.027320 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.027909 (+ 589us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.028059 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:29.998699 4057 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:29.946722 3922 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:30.045560 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:30.047093 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1392) took 3516 ms. Trace:
I20260927 23:31:30.046036 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 23:31:30.034030 4204 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.346s user 0.001s sys 0.000s
I20260927 23:31:30.060488 3763 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.060616 1627 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:30.046551 1732 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:30.060508 4202 log.cc:924] Time spent T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260927 23:31:30.047554 1579 rpcz_store.cc:276] 0927 23:31:26.530824 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.184694 (+1653870us) service_pool.cc:227] Handling call
0927 23:31:28.423657 (+238963us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b5800001898. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:28.423872 (+ 215us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.431536 (+ 7664us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.431579 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.476376 (+ 44797us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.868818 (+392442us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:28.868868 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.868950 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.039476 (+1170526us) raft_consensus.cc:1671] finished
0927 23:31:30.039504 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.047017 (+ 7513us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":674,"raft.queue_time_us":7861,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":23642,"spinlock_wait_cycles":74827008,"thread_start_us":319135,"threads_started":1,"wal-append.queue_time_us":816247}
I20260927 23:31:30.092494 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1434) took 1824 ms. Trace:
W20260927 23:31:30.091921 1700 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.004s sys 0.012s
W20260927 23:31:30.093253 1700 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.004s sys 0.012s
W20260927 23:31:30.093972 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 643546us
I20260927 23:31:30.095427 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 371) took 3178 ms. Trace:
I20260927 23:31:30.095979 1584 rpcz_store.cc:276] 0927 23:31:26.916893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.029759 (+3112866us) service_pool.cc:227] Handling call
0927 23:31:30.083419 (+ 53660us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.095115 (+ 11696us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.095139 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.095180 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:30.095224 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.095279 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.095377 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:30.094558 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:30.098057 1728 raft_consensus.cc:2396] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260927 23:31:30.102931 2287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 512) took 1929 ms. Trace:
I20260927 23:31:30.104008 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:30.105146 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 44644.8us
I20260927 23:31:30.093047 1737 rpcz_store.cc:276] 0927 23:31:28.268363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.268532 (+ 169us) service_pool.cc:227] Handling call
0927 23:31:28.311053 (+ 42521us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:28.432254 (+121201us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.432295 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:28.515242 (+ 82947us) log.cc:842] Serialized 52 byte log entry
0927 23:31:28.753972 (+238730us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:29.616172 (+862200us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b18002e2c40. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47764 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:29.776936 (+160764us) log.cc:842] Serialized 14 byte log entry
0927 23:31:30.063705 (+286769us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.063774 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.063818 (+ 44us) raft_consensus.cc:1671] finished
0927 23:31:30.083694 (+ 19876us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.092410 (+ 8716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57207,"prepare.queue_time_us":743133,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":233319,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":24488,"spinlock_wait_cycles":45193984,"thread_start_us":1332550,"threads_started":3,"wal-append.queue_time_us":1048590}
I20260927 23:31:30.108965 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:29.944514 4201 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.074501 3922 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:30.115288 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 373) took 3170 ms. Trace:
I20260927 23:31:30.118789 1583 rpcz_store.cc:276] 0927 23:31:26.944583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.038975 (+3094392us) service_pool.cc:227] Handling call
0927 23:31:30.080446 (+ 41471us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.108834 (+ 28388us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.108858 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.114997 (+ 6139us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:30.115062 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.115133 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.115237 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:30.120838 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1570) took 1449 ms. Trace:
W20260927 23:31:30.121133 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 94179.4us
I20260927 23:31:30.122394 1587 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:30.121218 1657 rpcz_store.cc:276] 0927 23:31:28.671764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.674465 (+ 2701us) service_pool.cc:227] Handling call
0927 23:31:30.103911 (+1429446us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0fc6c 00007f974db0fbd6 000000000034aea9 00007f975593ba30 00007f9755a017b4 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:30.120785 (+ 16874us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10793216}
I20260927 23:31:30.123027 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1484) took 3121 ms. Trace:
I20260927 23:31:30.124423 1587 rpcz_store.cc:276] 0927 23:31:27.001266 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.039597 (+3038331us) service_pool.cc:227] Handling call
0927 23:31:30.100560 (+ 60963us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.122787 (+ 22227us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.122975 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:30.125406 1700 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:30.113212 4137 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.518s user 0.001s sys 0.012s
I20260927 23:31:30.112291 4205 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:30.135776 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 188227us
W20260927 23:31:30.136963 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 191571us
I20260927 23:31:30.118103 4125 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.125257 3922 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
W20260927 23:31:30.140749 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 283622us
I20260927 23:31:30.103397 2287 rpcz_store.cc:276] 0927 23:31:28.173136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.175508 (+ 2372us) service_pool.cc:227] Handling call
0927 23:31:28.190761 (+ 15253us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.191036 (+ 275us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.191068 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.191596 (+ 528us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.316740 (+125144us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:28.392433 (+ 75693us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.398250 (+ 5817us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.074822 (+1676572us) raft_consensus.cc:1671] finished
0927 23:31:30.074854 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.102834 (+ 27980us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":115111,"threads_started":1,"wal-append.queue_time_us":1842492}
I20260927 23:31:30.116034 4069 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.123291 4206 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:30.128604 4057 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:30.110040 3776 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20260927 23:31:30.146263 1702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.007s sys 0.005s
W20260927 23:31:30.150085 2258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.006s sys 0.010s
I20260927 23:31:30.093657 3431 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.134927 4205 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:30.154963 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1433) took 1919 ms. Trace:
I20260927 23:31:30.157414 1734 rpcz_store.cc:276] 0927 23:31:28.236437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.239369 (+ 2932us) service_pool.cc:227] Handling call
0927 23:31:28.293641 (+ 54272us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.414186 (+120545us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.414233 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.443500 (+ 29267us) log.cc:842] Serialized 26 byte log entry
0927 23:31:29.368306 (+924806us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:29.368361 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.368517 (+ 156us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.154637 (+786120us) raft_consensus.cc:1671] finished
0927 23:31:30.154686 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.154885 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":689394,"threads_started":1,"wal-append.queue_time_us":1195050}
W20260927 23:31:30.153299 1702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.007s sys 0.005s
I20260927 23:31:30.138005 4130 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:30.160420 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:30.164549 2258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s user 0.006s sys 0.010s
W20260927 23:31:30.152284 4211 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
W20260927 23:31:30.171581 1585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.029s sys 0.020s
W20260927 23:31:30.172616 1585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.029s sys 0.020s
I20260927 23:31:30.152405 4069 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:30.174196 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.175774 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:30.162360 4205 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:30.162657 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:30.193073 1628 leader_election.cc:334] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.245s (SENT)
I20260927 23:31:30.194428 4214 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:30.193723 1628 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:30.195119 4214 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.192385 4069 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.197817 4233 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:30.198689 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:30.200347 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.200974 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1438) took 1657 ms. Trace:
I20260927 23:31:30.201495 1735 rpcz_store.cc:276] 0927 23:31:28.543855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.544010 (+ 155us) service_pool.cc:227] Handling call
0927 23:31:28.576704 (+ 32694us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:28.694393 (+117689us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:29.042350 (+347957us) log.cc:842] Serialized 14 byte log entry
0927 23:31:29.938427 (+896077us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:29.944253 (+ 5826us) log.cc:842] Serialized 52 byte log entry
0927 23:31:29.944382 (+ 129us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:29.947253 (+ 2871us) log.cc:842] Serialized 14 byte log entry
0927 23:31:29.947580 (+ 327us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.947678 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.200712 (+253034us) raft_consensus.cc:1671] finished
0927 23:31:30.200738 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.200907 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62460,"prepare.queue_time_us":773237,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":111,"raft.queue_time_us":169012,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":15820,"thread_start_us":786178,"threads_started":2,"wal-append.queue_time_us":724021}
W20260927 23:31:30.245154 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 55369.9us
W20260927 23:31:30.255365 1628 leader_election.cc:334] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.839s (SENT)
I20260927 23:31:30.192700 4057 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:30.186676 4223 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:30.228915 2257 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.2:42051 timed out after 2.016s (SENT) [suppressed 33 similar messages]
W20260927 23:31:30.264278 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 846548us
I20260927 23:31:30.272051 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.163254 4206 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:30.262297 4057 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:30.288077 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 157008us
I20260927 23:31:30.303030 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:30.313647 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:30.314664 1729 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:30.297258 4057 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:30.282111 4227 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:30.362655 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:30.366688 2291 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:30.320894 4214 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.322064 4069 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:30.382133 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 61499.9us
I20260927 23:31:30.362414 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:30.371104 2287 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:30.390749 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.399628 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:30.400153 1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1571) took 1728 ms (client timeout 1930 ms). Trace:
W20260927 23:31:30.400595 1663 rpcz_store.cc:269] 0927 23:31:28.672148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.674895 (+ 2747us) service_pool.cc:227] Handling call
0927 23:31:30.375387 (+1700492us) spinlock_profiling.cc:243] Waited 968 us on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:30.400084 (+ 24697us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2226560}
I20260927 23:31:30.400542 2285 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:30.435201 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:30.435745 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 131026us
I20260927 23:31:30.441445 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:30.401245 2290 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:30.462226 1700 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:30.371160 4227 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.650183 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1489) took 3647 ms. Trace:
W20260927 23:31:30.550793 1661 consensus_meta.cc:287] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.613s user 0.010s sys 0.001s
I20260927 23:31:30.668252 1661 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:30.525372 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:30.597242 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 107946us
W20260927 23:31:30.619506 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 157379us
W20260927 23:31:30.662839 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 224406us
I20260927 23:31:30.665966 1579 rpcz_store.cc:276] 0927 23:31:27.002875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.082425 (+3079550us) service_pool.cc:227] Handling call
0927 23:31:30.121133 (+ 38708us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:30.137510 (+ 16377us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.137554 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:30.153607 (+ 16053us) log.cc:842] Serialized 26 byte log entry
0927 23:31:30.272258 (+118651us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:30.311706 (+ 39448us) log.cc:842] Serialized 14 byte log entry
0927 23:31:30.311979 (+ 273us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.312058 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.649907 (+337849us) raft_consensus.cc:1671] finished
0927 23:31:30.649935 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.650102 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":169659,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":6816,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":9685,"thread_start_us":104237,"threads_started":2,"wal-append.queue_time_us":272319}
I20260927 23:31:30.681562 1729 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260927 23:31:30.674288 1702 leader_election.cc:334] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.661s (SENT)
I20260927 23:31:30.683189 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:30.686174 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1b945905616b4989853b0a92bfbd6b98 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:30.571061 4195 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.963s user 0.007s sys 0.015s
I20260927 23:31:30.689842 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.694245 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:30.694928 1628 leader_election.cc:334] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.514s (SENT)
I20260927 23:31:30.625403 4214 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260927 23:31:30.707230 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 336426us
I20260927 23:31:30.689038 2289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 514) took 1318 ms. Trace:
I20260927 23:31:30.735095 2289 rpcz_store.cc:276] 0927 23:31:29.367199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.376845 (+ 9646us) service_pool.cc:227] Handling call
0927 23:31:29.551878 (+175033us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:29.760118 (+208240us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:29.767481 (+ 7363us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:29.886272 (+118791us) log.cc:842] Serialized 26 byte log entry
0927 23:31:30.160858 (+274586us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:30.161417 (+ 559us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.168203 (+ 6786us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.662745 (+494542us) raft_consensus.cc:1671] finished
0927 23:31:30.664852 (+ 2107us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.685339 (+ 20487us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1146,"raft.queue_time_us":76745,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":47,"thread_start_us":149067,"threads_started":1,"wal-append.queue_time_us":360383}
I20260927 23:31:30.736263 4207 consensus_queue.cc:1046] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.300s
I20260927 23:31:30.523128 4069 consensus_queue.cc:1035] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.253s
I20260927 23:31:30.704949 4229 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:30.743316 4229 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:30.743598 2290 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
W20260927 23:31:30.745558 4136 consensus_meta.cc:287] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.420s user 0.016s sys 0.006s
I20260927 23:31:30.746140 4136 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:30.748622 4136 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:30.749421 1654 raft_consensus.cc:1226] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
W20260927 23:31:30.760025 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 88566.8us
I20260927 23:31:30.679751 3952 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260927 23:31:30.776928 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.718801 4214 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:30.833474 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.839049 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.713379 4195 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.055065 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1580) took 2263 ms. Trace:
I20260927 23:31:31.423460 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:30.811223 4214 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:30.969697 1656 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260927 23:31:31.143610 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:31.174219 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:30.812815 3431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.000s sys 0.013s
W20260927 23:31:31.154971 1703 leader_election.cc:334] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Timed out: RequestConsensusVote RPC to 127.0.99.4:45509 timed out after 1.845s (SENT)
I20260927 23:31:31.173506 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:30.898712 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1393) took 4332 ms. Trace:
I20260927 23:31:31.435976 1703 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:31.429069 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 293) took 1214 ms. Trace:
W20260927 23:31:30.862861 1658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1577) took 2166 ms (client timeout 1639 ms). Trace:
I20260927 23:31:31.173112 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:31.053005 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:31.157764 1737 raft_consensus.cc:1261] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:31.156867 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:31.441368 1728 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 3.
I20260927 23:31:31.423739 1659 rpcz_store.cc:276] 0927 23:31:28.791689 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.791862 (+ 173us) service_pool.cc:227] Handling call
0927 23:31:31.054052 (+2262190us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:31.054295 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.054610 (+ 315us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:31.054643 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.054721 (+ 78us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:31.054780 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.054852 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.054978 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309483904}
I20260927 23:31:31.211002 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 369) took 4371 ms. Trace:
W20260927 23:31:31.397948 1472 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 486149us
I20260927 23:31:31.443401 1581 rpcz_store.cc:276] 0927 23:31:26.839892 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.024867 (+3184975us) service_pool.cc:227] Handling call
0927 23:31:30.025197 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.025432 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:30.301984 (+276552us) log.cc:842] Serialized 14 byte log entry
0927 23:31:31.175374 (+873390us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.175447 (+ 73us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:31.210676 (+ 35229us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.210774 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.210931 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":376041,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":1078447,"threads_started":3}
I20260927 23:31:31.057798 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1586) took 1452 ms. Trace:
I20260927 23:31:31.437117 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1774) took 1164 ms. Trace:
I20260927 23:31:31.446354 1655 rpcz_store.cc:276] 0927 23:31:29.605096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.988079 (+382983us) service_pool.cc:227] Handling call
0927 23:31:30.002734 (+ 14655us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.059747 (+ 57013us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.059784 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.059829 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:30.059867 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.057478 (+997611us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54000dc0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:31.057537 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.057730 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94029184}
W20260927 23:31:31.437278 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 774867us
I20260927 23:31:31.436555 1656 rpcz_store.cc:276] 0927 23:31:30.214426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.911954 (+697528us) service_pool.cc:227] Handling call
0927 23:31:30.963024 (+ 51070us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.428831 (+465807us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.429019 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:31.445356 1661 rpcz_store.cc:276] 0927 23:31:30.272536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:31.436432 (+1163896us) service_pool.cc:227] Handling call
0927 23:31:31.436697 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.436863 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:31.436884 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.436920 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:31.436951 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.436994 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.437074 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:31.439550 1729 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:31.455555 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46200 (request call id 1575) took 2782 ms. Trace:
W20260927 23:31:31.050652 4182 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.818s user 0.009s sys 0.015s
W20260927 23:31:31.247741 4252 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
W20260927 23:31:31.438578 1658 rpcz_store.cc:269] 0927 23:31:28.674029 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.675643 (+ 1614us) service_pool.cc:227] Handling call
0927 23:31:30.838847 (+2163204us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:30.840066 (+ 1219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2308992}
I20260927 23:31:31.456916 1734 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:31.456975 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 776418us
I20260927 23:31:31.456067 1662 rpcz_store.cc:276] 0927 23:31:28.673387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.675191 (+ 1804us) service_pool.cc:227] Handling call
0927 23:31:31.052800 (+2377609us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b5800004598. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:31.455502 (+402702us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2305920}
W20260927 23:31:31.458074 1735 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.013s sys 0.037s
W20260927 23:31:31.481220 1735 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.764s user 0.020s sys 0.048s
I20260927 23:31:31.476231 1588 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 8. Current term is 9. Ops: []
W20260927 23:31:31.054235 4255 log.cc:924] Time spent T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260927 23:31:31.435385 3431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.221s user 0.007s sys 0.039s
W20260927 23:31:31.457684 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 556670us
I20260927 23:31:31.482910 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1775) took 1179 ms. Trace:
W20260927 23:31:30.962126 4237 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.244s user 0.001s sys 0.000s
W20260927 23:31:31.450852 4221 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.005s sys 0.021s
W20260927 23:31:31.486021 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 342479us
W20260927 23:31:31.487905 1628 leader_election.cc:334] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.528s (SENT)
I20260927 23:31:30.863210 4206 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:31.428004 4227 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.502480 1734 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:31.434139 4130 raft_consensus.cc:3032] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:31.493289 1659 rpcz_store.cc:276] 0927 23:31:30.303451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:31.442870 (+1139419us) service_pool.cc:227] Handling call
0927 23:31:31.482218 (+ 39348us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.482461 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:31.482497 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.482551 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:31.482626 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.482694 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.482842 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:31.494246 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:30.939510 4205 consensus_meta.cc:287] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.741s user 0.014s sys 0.001s
I20260927 23:31:31.436296 1585 rpcz_store.cc:276] 0927 23:31:26.531228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.717823 (+2186595us) service_pool.cc:227] Handling call
0927 23:31:28.845367 (+127544us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:29.545434 (+700067us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:29.545574 (+ 140us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:29.570784 (+ 25210us) log.cc:842] Serialized 26 byte log entry
0927 23:31:30.173051 (+602267us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:30.320356 (+147305us) log.cc:842] Serialized 14 byte log entry
0927 23:31:30.320668 (+ 312us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:30.320751 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:30.863007 (+542256us) raft_consensus.cc:1671] finished
0927 23:31:30.863032 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:30.863198 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11523,"prepare.queue_time_us":268614,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":448137,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":28347,"thread_start_us":1361081,"threads_started":3,"wal-append.queue_time_us":686493}
W20260927 23:31:31.511279 1702 leader_election.cc:334] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.969s (SENT)
I20260927 23:31:31.511922 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:31.485487 4216 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.005s sys 0.019s
I20260927 23:31:31.493788 4201 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:30.859805 4229 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:31.508602 3763 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.029s sys 0.024s
I20260927 23:31:31.496573 4217 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:31.531950 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 377014us
I20260927 23:31:31.535702 4228 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:31.533686 4249 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260927 23:31:31.507490 4130 raft_consensus.cc:729] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:31.535360 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:31.508759 4206 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:31.530386 4216 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.005s sys 0.020s
W20260927 23:31:31.536124 4185 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 2.303s user 0.004s sys 0.010s
I20260927 23:31:31.534265 4227 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:31.533305 3431 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:31.508920 4205 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:31.576716 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 96842.1us
W20260927 23:31:31.544620 3763 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.029s sys 0.024s
I20260927 23:31:31.606033 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 291) took 1392 ms. Trace:
I20260927 23:31:31.552309 4266 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:31.606453 1660 rpcz_store.cc:276] 0927 23:31:30.213862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.300246 (+ 86384us) service_pool.cc:227] Handling call
0927 23:31:30.357409 (+ 57163us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:30.661256 (+303847us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.661296 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:30.674123 (+ 12827us) log.cc:842] Serialized 26 byte log entry
0927 23:31:31.162217 (+488094us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:31.162254 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.162362 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.605788 (+443426us) raft_consensus.cc:1671] finished
0927 23:31:31.605809 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.605936 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9523,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5357,"thread_start_us":423222,"threads_started":1,"wal-append.queue_time_us":893911}
W20260927 23:31:31.607972 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 96723.5us
I20260927 23:31:31.558670 4206 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:31.607883 1628 leader_election.cc:334] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.593s (SENT)
W20260927 23:31:31.612469 3922 consensus_meta.cc:287] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 1.459s user 0.013s sys 0.004s
W20260927 23:31:31.613317 4253 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260927 23:31:31.613006 3922 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.550274 4217 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:31.616072 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:31.617022 1663 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:31.616268 4029 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:31.599222 4265 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:31.618688 4205 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:31.549595 4228 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:31.653750 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1490) took 4650 ms. Trace:
I20260927 23:31:31.622284 4265 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:31.660915 4184 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 1.679s user 0.012s sys 0.008s
I20260927 23:31:31.658707 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 379) took 2099 ms. Trace:
I20260927 23:31:31.668550 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1390) took 5159 ms. Trace:
I20260927 23:31:31.663404 1583 rpcz_store.cc:276] 0927 23:31:29.559473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.124191 (+564718us) service_pool.cc:227] Handling call
0927 23:31:30.132561 (+ 8370us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.138024 (+ 5463us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:30.286168 (+148144us) log.cc:842] Serialized 14 byte log entry
0927 23:31:31.658265 (+1372097us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.658361 (+ 96us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:31.658428 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.658511 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.658637 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":635386,"prepare.queue_time_us":281201,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":768849,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":779151,"threads_started":3,"wal-append.queue_time_us":1245620}
I20260927 23:31:31.651046 4205 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:31.600416 3431 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.669070 1586 rpcz_store.cc:276] 0927 23:31:26.509122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.213367 (+1704245us) service_pool.cc:227] Handling call
0927 23:31:28.223101 (+ 9734us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.241936 (+ 18835us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.241981 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.292486 (+ 50505us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.442916 (+150430us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:28.489064 (+ 46148us) log.cc:842] Serialized 14 byte log entry
0927 23:31:28.683255 (+194191us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.683417 (+ 162us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.660363 (+2976946us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:31.662387 (+ 2024us) raft_consensus.cc:1671] finished
0927 23:31:31.668333 (+ 5946us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.668478 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4097,"prepare.queue_time_us":46234,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2731,"raft.queue_time_us":35054,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":35940,"spinlock_wait_cycles":27278208,"thread_start_us":42729,"threads_started":1,"wal-append.queue_time_us":1521335}
I20260927 23:31:31.653081 3763 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:31.674652 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75207.4us
W20260927 23:31:31.697575 1628 leader_election.cc:334] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.618s (SENT)
I20260927 23:31:31.697986 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:31.619225 4206 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:31.706836 4274 log.cc:924] Time spent T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260927 23:31:31.708320 4257 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260927 23:31:31.664094 4265 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:31.714546 4205 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:31.714242 3431 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:31.714632 4270 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260927 23:31:31.717831 4206 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.713963 2288 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 480) took 1438 ms. Trace:
I20260927 23:31:31.722550 2288 rpcz_store.cc:276] 0927 23:31:30.275398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.276546 (+ 1148us) service_pool.cc:227] Handling call
0927 23:31:31.713203 (+1436657us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559c52b0 00007f97559b07f9 00007f97559b02a2 00007f974f096684
0927 23:31:31.713406 (+ 203us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.713607 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:31.713639 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:31.713706 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:31.713743 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.713799 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.713906 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86943104}
I20260927 23:31:31.724011 4276 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.724781 4276 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.727711 4228 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:31.729151 4217 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:31.655107 4242 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260927 23:31:31.729497 4276 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.615113 4266 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:31.730491 4269 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:31.730862 4269 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:31.731992 4266 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:31.731807 4269 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:31.732795 4229 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:31.733073 4269 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:31.733393 4276 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.733918 4276 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.699505 4195 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.719028 3763 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.735316 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:31.742936 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1432) took 3544 ms. Trace:
I20260927 23:31:31.743275 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 362) took 1437 ms. Trace:
I20260927 23:31:31.762214 1731 rpcz_store.cc:276] 0927 23:31:28.198308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.204898 (+ 6590us) service_pool.cc:227] Handling call
0927 23:31:28.205203 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:28.205751 (+ 548us) raft_consensus.cc:1520] Early marking committed up to index 1
0927 23:31:28.205786 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:28.206239 (+ 453us) log.cc:842] Serialized 26 byte log entry
0927 23:31:28.537172 (+330933us) raft_consensus.cc:1631] Marking committed up to 1
0927 23:31:28.537300 (+ 128us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.537451 (+ 151us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.414548 (+2877097us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:31.702981 (+288433us) raft_consensus.cc:1671] finished
0927 23:31:31.703010 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.742871 (+ 39861us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":168,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":84223232,"thread_start_us":300453,"threads_started":1,"wal-append.queue_time_us":1026758}
I20260927 23:31:31.762389 1735 rpcz_store.cc:276] 0927 23:31:30.306005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.306126 (+ 121us) service_pool.cc:227] Handling call
0927 23:31:30.360010 (+ 53884us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:30.360488 (+ 478us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:30.360530 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:30.379772 (+ 19242us) log.cc:842] Serialized 26 byte log entry
0927 23:31:30.716823 (+337051us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:31.486632 (+769809us) log.cc:842] Serialized 14 byte log entry
0927 23:31:31.742883 (+256251us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.742981 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.743029 (+ 48us) raft_consensus.cc:1671] finished
0927 23:31:31.743047 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.743209 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9541,"prepare.queue_time_us":951117,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":24550,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":18623,"thread_start_us":1307970,"threads_started":3,"wal-append.queue_time_us":417661}
W20260927 23:31:31.736348 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 975319us
I20260927 23:31:31.761710 1737 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:31.644059 4130 consensus_queue.cc:260] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.787055 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:31.787954 1731 raft_consensus.cc:2396] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260927 23:31:31.787968 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 1.04745e+06us
I20260927 23:31:31.786442 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 284) took 1627 ms. Trace:
W20260927 23:31:31.769753 4057 consensus_meta.cc:287] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.393s user 0.013s sys 0.001s
I20260927 23:31:31.762099 4242 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.765806 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:31.799799 1735 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:31.795017 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:31.795925 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:31.803277 1660 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:31.790052 1657 rpcz_store.cc:276] 0927 23:31:30.158809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.159392 (+ 583us) service_pool.cc:227] Handling call
0927 23:31:30.170368 (+ 10976us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.199740 (+ 29372us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:30.287517 (+ 87777us) log.cc:842] Serialized 14 byte log entry
0927 23:31:30.376652 (+ 89135us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:30.376716 (+ 64us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:30.376784 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.786073 (+1409289us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b54002f3ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:31.786138 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.786373 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":61251,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":9044,"raft.queue_time_us":2798,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":8462,"spinlock_wait_cycles":66878208,"thread_start_us":176137,"threads_started":2,"wal-append.queue_time_us":447478}
I20260927 23:31:31.769259 4195 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.795331 3900 raft_consensus.cc:3032] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:31.882920 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:31.895720 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 180150us
I20260927 23:31:31.868834 4229 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.800695 4057 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.884554 3900 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:31.886451 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:31.883278 4266 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:31.899452 4214 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.937502 4277 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.157s
I20260927 23:31:31.942868 4275 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:31.946830 4186 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 2.720s user 0.011s sys 0.018s
W20260927 23:31:31.947705 4179 log.cc:924] Time spent T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 3.386s user 0.020s sys 0.049s
I20260927 23:31:31.951879 4276 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.969000 4040 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:31.969328 4276 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:31.969717 4214 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.969628 4040 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.969846 4276 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.971645 4040 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:31.972150 4276 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.973986 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1431) took 3776 ms. Trace:
I20260927 23:31:31.974401 1736 rpcz_store.cc:276] 0927 23:31:28.197868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.203232 (+ 5364us) service_pool.cc:227] Handling call
0927 23:31:28.203571 (+ 339us) raft_consensus.cc:1457] Updating replica for 3 ops
0927 23:31:28.204491 (+ 920us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:28.204519 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0927 23:31:28.205211 (+ 692us) log.cc:842] Serialized 78 byte log entry
0927 23:31:28.521676 (+316465us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:28.709632 (+187956us) log.cc:842] Serialized 14 byte log entry
0927 23:31:28.709933 (+ 301us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:28.710032 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.864274 (+3154242us) spinlock_profiling.cc:243] Waited 8.43 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:31.973551 (+109277us) raft_consensus.cc:1671] finished
0927 23:31:31.973678 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.973920 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49982,"prepare.queue_time_us":187652,"prepare.run_cpu_time_us":5274,"prepare.run_wall_time_us":24656,"raft.queue_time_us":6174,"raft.run_cpu_time_us":881,"raft.run_wall_time_us":31993,"spinlock_wait_cycles":19395968,"thread_start_us":316035,"threads_started":1,"wal-append.queue_time_us":1021107}
I20260927 23:31:31.901726 4050 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:31.874545 4242 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.977617 4050 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.979228 4242 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.979391 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:31.962734 1733 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 23:31:31.966534 4200 raft_consensus.cc:3032] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:31.966600 3763 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:31.966349 4275 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.884267 4195 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.979878 4050 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:32.010804 1580 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:32.016208 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1526) took 3794 ms. Trace:
I20260927 23:31:32.016716 1582 rpcz_store.cc:276] 0927 23:31:28.193090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.703541 (+510451us) service_pool.cc:227] Handling call
0927 23:31:31.982505 (+3278964us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b5400466754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:31.987017 (+ 4512us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1381825408}
I20260927 23:31:31.902094 4229 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:31.972144 4040 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:32.031693 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:31.985234 4200 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:32.032954 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.036850 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.037405 1586 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:32.038605 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 155789us
I20260927 23:31:32.041756 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.044826 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.044857 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:31.980165 2286 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:31.962405 4266 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.015575 3763 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:32.026960 4040 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:32.062450 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 194011us
I20260927 23:31:32.063060 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.067144 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.068192 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:31.990247 1735 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:32.074158 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.078019 1659 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
W20260927 23:31:32.084122 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 183570us
I20260927 23:31:32.092546 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.093816 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.094604 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.095324 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.095664 4266 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:32.035689 3900 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.081055 4040 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:32.092473 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:32.093891 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:32.108160 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.106590 1734 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:32.107565 1659 raft_consensus.cc:2362] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7. Re-sending same reply.
I20260927 23:31:32.112957 2257 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:32.113083 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:32.114780 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:32.119025 4285 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
W20260927 23:31:32.084970 2286 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51538 (request call id 484) took 1799 ms (client timeout 1845 ms). Trace:
I20260927 23:31:32.132102 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:32.133127 1660 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:32.083986 3763 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:32.146292 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.115546 3900 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:32.115767 4057 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:32.167624 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:32.121426 4286 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:32.170447 1552 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:32.181459 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1773) took 1924 ms. Trace:
I20260927 23:31:32.190269 1663 rpcz_store.cc:276] 0927 23:31:30.256636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:31.434814 (+1178178us) service_pool.cc:227] Handling call
0927 23:31:31.435148 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.620981 (+185833us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:32.181172 (+560191us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000bb3d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:32.181388 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411662080}
I20260927 23:31:32.131028 4292 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:32.135229 4276 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:32.215456 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.145530 4279 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:32.178819 1628 connection.cc:232] RPC call timeout handler was delayed by -1.60339717100004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2649s for a response
W20260927 23:31:32.224009 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 114641us
I20260927 23:31:32.212863 4195 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:32.128522 2286 rpcz_store.cc:269] 0927 23:31:30.278202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.280212 (+ 2010us) service_pool.cc:227] Handling call
0927 23:31:31.979217 (+1699005us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0f851 00007f974db0f702 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:32.073784 (+ 94567us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87055232}
I20260927 23:31:32.231444 1627 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:32.104977 4200 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.215794 4286 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.223901 4279 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.258644 4291 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:32.266671 1628 leader_election.cc:334] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.970s (SENT)
I20260927 23:31:32.267266 1628 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:32.266090 1700 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:32.267616 4267 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:32.273502 4267 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:32.274418 4299 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:32.274892 4299 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:32.275291 4299 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:32.277941 4297 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:32.278573 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 108288us
I20260927 23:31:32.296416 4229 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:32.289496 4200 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:32.260260 4276 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:32.301476 4265 consensus_meta.cc:287] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.001s
I20260927 23:31:32.278033 4040 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.307803 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.315908 4265 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:32.318475 4289 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.273s user 0.001s sys 0.001s
I20260927 23:31:32.317830 4310 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:32.317319 4301 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:32.318807 4307 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:32.314797 4287 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:32.315222 4309 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260927 23:31:32.301819 4275 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:32.330788 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 73460.5us
I20260927 23:31:32.333854 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:32.339466 4307 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:32.255479 4292 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:32.278700 4297 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:32.343173 1657 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260927 23:31:32.315236 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:32.364415 1628 leader_election.cc:334] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.149s (SENT)
W20260927 23:31:32.366855 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 168032us
W20260927 23:31:32.365381 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 71879.5us
I20260927 23:31:32.372948 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:32.376039 1628 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:32.405068 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.291154 4291 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:29.942829 1730 rpcz_store.cc:276] 0927 23:31:28.460458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.460635 (+ 177us) service_pool.cc:227] Handling call
0927 23:31:28.489529 (+ 28894us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:28.561695 (+ 72166us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:28.870417 (+308722us) log.cc:842] Serialized 14 byte log entry
0927 23:31:29.889326 (+1018909us) log.cc:842] Serialized 14 byte log entry
0927 23:31:29.937620 (+ 48294us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:29.937678 (+ 58us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:29.941381 (+ 3703us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:29.941470 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:29.941586 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":136202,"prepare.queue_time_us":308631,"prepare.run_cpu_time_us":83488,"prepare.run_wall_time_us":449368,"raft.queue_time_us":394089,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":20723,"thread_start_us":764524,"threads_started":3,"wal-append.queue_time_us":633399}
I20260927 23:31:32.371356 4287 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:32.421696 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:32.406755 4301 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 23:31:32.421253 4284 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.253s user 0.001s sys 0.000s
I20260927 23:31:32.417085 4195 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:32.424165 4296 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.214s user 0.000s sys 0.001s
I20260927 23:31:32.500437 4040 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:32.503054 4279 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:32.523352 4269 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.790s user 0.011s sys 0.002s
W20260927 23:31:32.523895 4302 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
W20260927 23:31:32.524426 4305 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
W20260927 23:31:32.548038 4311 log.cc:924] Time spent T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20260927 23:31:32.503808 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1419) took 5957 ms. Trace:
I20260927 23:31:32.506016 4265 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:32.424887 4069 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:32.506276 4312 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260927 23:31:32.507059 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1413) took 5970 ms. Trace:
I20260927 23:31:32.485989 4301 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:32.565172 1580 rpcz_store.cc:276] 0927 23:31:26.537014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:28.790252 (+2253238us) service_pool.cc:227] Handling call
0927 23:31:31.982709 (+3192457us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:32.015060 (+ 32351us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:32.506888 (+491828us) spinlock_profiling.cc:243] Waited 3.09 s on lock 0x7b5400466750. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:32.507010 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7101815168}
I20260927 23:31:32.552042 1588 rpcz_store.cc:276] 0927 23:31:26.545971 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.367554 (+2821583us) service_pool.cc:227] Handling call
0927 23:31:29.532908 (+165354us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:31.484710 (+1951802us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:32.503500 (+1018790us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b540009f6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:32.503732 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4468751616}
I20260927 23:31:32.566948 1660 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:32.567528 4208 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:32.570923 4306 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.240s user 0.000s sys 0.001s
W20260927 23:31:32.553709 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 893376us
I20260927 23:31:32.436771 3763 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:32.574307 4287 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:32.592530 4307 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:32.595587 4304 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.336s user 0.003s sys 0.000s
W20260927 23:31:32.646958 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 227412us
I20260927 23:31:32.592482 4301 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:32.659687 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 337823us
W20260927 23:31:32.660941 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 316903us
I20260927 23:31:31.655428 1584 rpcz_store.cc:276] 0927 23:31:27.003142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.097138 (+3093996us) service_pool.cc:227] Handling call
0927 23:31:30.133383 (+ 36245us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:30.163578 (+ 30195us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 23:31:30.277470 (+113892us) log.cc:842] Serialized 14 byte log entry
0927 23:31:30.751566 (+474096us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:31.456722 (+705156us) log.cc:842] Serialized 26 byte log entry
0927 23:31:31.479854 (+ 23132us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:31.480198 (+ 344us) log.cc:842] Serialized 14 byte log entry
0927 23:31:31.653225 (+173027us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:31.653325 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:31.653384 (+ 59us) raft_consensus.cc:1671] finished
0927 23:31:31.653404 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:31.653563 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6495,"prepare.queue_time_us":344096,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":1605,"raft.queue_time_us":1114908,"raft.run_cpu_time_us":389,"raft.run_wall_time_us":17431,"spinlock_wait_cycles":940928,"thread_start_us":639711,"threads_started":4,"wal-append.queue_time_us":628393}
W20260927 23:31:32.661757 1628 leader_election.cc:334] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.191s (SENT)
I20260927 23:31:32.669476 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:32.684170 4195 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:32.696368 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1460) took 1343 ms. Trace:
I20260927 23:31:32.696812 1729 rpcz_store.cc:276] 0927 23:31:31.353474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:31.520980 (+167506us) service_pool.cc:227] Handling call
0927 23:31:31.545804 (+ 24824us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:31.883860 (+338056us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:31.883901 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:31.899047 (+ 15146us) log.cc:842] Serialized 26 byte log entry
0927 23:31:32.074020 (+174973us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:32.074053 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:32.696002 (+621949us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b540011c454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:32.696042 (+ 40us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:32.696086 (+ 44us) raft_consensus.cc:1671] finished
0927 23:31:32.696099 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:32.696307 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":58782,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"spinlock_wait_cycles":7399168,"thread_start_us":441548,"threads_started":2,"wal-append.queue_time_us":203704}
I20260927 23:31:32.701815 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.703661 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:32.704870 2286 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:32.704908 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.703706 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.706064 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:32.792673 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 361) took 2487 ms. Trace:
W20260927 23:31:33.255051 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 591151us
W20260927 23:31:32.767876 1703 leader_election.cc:334] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Timed out: RequestConsensusVote RPC to 127.0.99.4:45509 timed out after 1.765s (SENT)
I20260927 23:31:32.706738 2291 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:33.256932 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 489083us
I20260927 23:31:33.252516 2258 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:33.254552 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 680929us
W20260927 23:31:33.252349 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 590570us
W20260927 23:31:32.828444 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 408048us
I20260927 23:31:33.256282 1733 rpcz_store.cc:276] 0927 23:31:30.305230 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:30.305665 (+ 435us) service_pool.cc:227] Handling call
0927 23:31:30.358116 (+ 52451us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:31.984892 (+1626776us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:32.792358 (+807466us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400105254. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:32.792609 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3683955584}
W20260927 23:31:33.266069 1628 leader_election.cc:334] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.215s (SENT)
I20260927 23:31:33.269500 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:32.849468 4295 log.cc:924] Time spent T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.907s user 0.001s sys 0.001s
W20260927 23:31:32.940397 4263 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.228s user 0.007s sys 0.013s
I20260927 23:31:33.273728 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:33.274731 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:33.233888 4299 consensus_meta.cc:287] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.958s user 0.001s sys 0.007s
W20260927 23:31:32.825172 4306 log.cc:924] Time spent T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260927 23:31:33.282006 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:33.282189 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:33.279623 1733 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:32.936030 4276 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:33.285934 1729 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:32.825122 4291 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:33.295754 1702 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:33.279403 4263 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.567s user 0.007s sys 0.014s
I20260927 23:31:33.283004 4277 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
W20260927 23:31:33.304497 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 45496us
W20260927 23:31:33.298581 4307 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.013s sys 0.042s
W20260927 23:31:33.313766 1628 leader_election.cc:334] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.217s (SENT)
I20260927 23:31:33.303301 2290 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:33.315526 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 613391us
I20260927 23:31:33.316151 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:33.311594 4307 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.014s sys 0.043s
I20260927 23:31:33.286600 4299 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.317656 4277 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:33.346634 1628 leader_election.cc:334] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.439s (SENT)
I20260927 23:31:33.352155 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:33.347985 3922 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.354571 4277 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:33.363667 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 77789.4us
W20260927 23:31:33.349584 4301 consensus_meta.cc:287] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.688s user 0.008s sys 0.000s
I20260927 23:31:33.362524 1729 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:33.322662 4325 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:33.370203 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:33.371136 2288 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 516) took 1575 ms (client timeout 2096 ms). Trace:
W20260927 23:31:33.371518 2288 rpcz_store.cc:269] 0927 23:31:31.795852 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:31.798605 (+ 2753us) service_pool.cc:227] Handling call
0927 23:31:33.369977 (+1571372us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b5800005198. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974db0fb7c 00007f974db0f73b 00007f97569ffb09 00007f975593b730 00007f9755a0179b 00007f9755a01821 00007f97559b11b0 00007f97559b0a94 00007f974f098c04 00007f974f098b92
0927 23:31:33.371070 (+ 1093us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18168576}
I20260927 23:31:33.373027 4299 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:33.373721 4299 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:33.374670 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:33.366467 4221 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:33.368364 4333 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:33.384848 1628 connection.cc:232] RPC call timeout handler was delayed by -1.2163971710000396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30790000000000006s for a response
I20260927 23:31:33.367247 4301 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.394770 1552 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.386898 4205 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:33.395711 1628 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:33.389238 4292 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:33.383442 4221 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.417209 1660 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:33.422551 4333 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:33.414567 4325 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.440161 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:33.443274 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 59931us
W20260927 23:31:33.430351 4287 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.024s sys 0.022s
I20260927 23:31:33.416668 4205 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.450667 4029 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:33.445195 4279 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:33.439826 4325 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:33.460111 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.444839 4292 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:33.448411 4287 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s user 0.025s sys 0.022s
W20260927 23:31:33.463855 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 200611us
W20260927 23:31:33.466537 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 70951.7us
W20260927 23:31:33.462571 4331 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260927 23:31:33.469655 4276 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.463629 4325 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.472957 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:33.475255 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1480) took 1053 ms. Trace:
W20260927 23:31:33.476583 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 210662us
I20260927 23:31:33.474249 4325 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:33.479282 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 159016us
I20260927 23:31:33.476511 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.475632 1737 rpcz_store.cc:276] 0927 23:31:32.421355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:32.421465 (+ 110us) service_pool.cc:227] Handling call
0927 23:31:32.421737 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:32.421906 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:32.438383 (+ 16477us) log.cc:842] Serialized 14 byte log entry
0927 23:31:32.555250 (+116867us) log.cc:842] Serialized 14 byte log entry
0927 23:31:33.474811 (+919561us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b1800299aa0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974dabe4ca 00007f975719ff99 00007f974d982990 00007f974db44864 00007f974db44155 00007f9750a673d0 00007f9750a79576 00007f9750a8a0d7 00007f9750a8a058
0927 23:31:33.474942 (+ 131us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:33.474984 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:33.475027 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:33.475091 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:33.475194 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47489,"prepare.queue_time_us":45885,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":19975,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":36567,"spinlock_wait_cycles":11646336}
W20260927 23:31:33.481112 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 210506us
I20260927 23:31:33.468942 4301 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.475067 2288 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:33.463593 4279 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.491230 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.492194 1657 raft_consensus.cc:2396] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 23:31:33.469774 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:33.458222 4029 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.483355 2258 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.495112 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1478) took 1075 ms. Trace:
I20260927 23:31:33.495998 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:33.501235 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.505024 4221 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:33.500027 4209 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:33.505264 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:33.505709 1552 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.500967 1732 rpcz_store.cc:276] 0927 23:31:32.420114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:32.420505 (+ 391us) service_pool.cc:227] Handling call
0927 23:31:32.486576 (+ 66071us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:32.486786 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:32.662757 (+175971us) log.cc:842] Serialized 14 byte log entry
0927 23:31:33.477090 (+814333us) log.cc:842] Serialized 14 byte log entry
0927 23:31:33.494788 (+ 17698us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:33.494840 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:33.494882 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:33.494944 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:33.495064 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":831389,"prepare.queue_time_us":42043,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":62,"raft.queue_time_us":816516,"spinlock_wait_cycles":4975872,"thread_start_us":136779,"threads_started":1}
I20260927 23:31:33.510474 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.510098 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:33.489614 4325 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 23:31:33.516242 4292 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.517690 4249 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:33.518787 2285 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 520) took 1455 ms. Trace:
I20260927 23:31:33.519287 2285 rpcz_store.cc:276] 0927 23:31:32.062946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:33.466987 (+1404041us) service_pool.cc:227] Handling call
0927 23:31:33.518703 (+ 51716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 23:31:33.520975 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.522497 1552 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.523475 4291 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:33.524290 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:33.524668 4342 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:33.525668 4342 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:33.525416 4263 consensus_queue.cc:1046] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:33.526229 4342 raft_consensus.cc:686] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:33.531564 4279 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.535851 4263 raft_consensus.cc:3032] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:33.536315 4263 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:33.537381 4263 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.529042 4342 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.541159 4263 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:33.551024 4205 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:33.505836 4209 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:33.564406 4334 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260927 23:31:33.565871 4130 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:33.566409 4217 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:33.567291 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.568065 4050 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:33.516628 4341 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:33.569528 4341 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:33.570377 4341 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.571460 4341 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:33.575241 4029 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:33.575500 4338 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260927 23:31:33.577456 4209 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.578435 2292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 457) took 3185 ms. Trace:
I20260927 23:31:33.598112 2292 rpcz_store.cc:276] 0927 23:31:30.392690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:32.699186 (+2306496us) service_pool.cc:227] Handling call
0927 23:31:32.699911 (+ 725us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:32.700415 (+ 504us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:32.700451 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:33.268735 (+568284us) log.cc:842] Serialized 480 byte log entry
0927 23:31:33.427489 (+158754us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:33.427540 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:33.427608 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:33.578224 (+150616us) raft_consensus.cc:1671] finished
0927 23:31:33.578244 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:33.578382 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":140223,"threads_started":1,"wal-append.queue_time_us":202146}
I20260927 23:31:33.526144 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.600431 1661 raft_consensus.cc:2396] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260927 23:31:33.582542 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 68154.8us
W20260927 23:31:33.603084 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 78928.9us
W20260927 23:31:33.583159 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 67036us
I20260927 23:31:33.604971 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.605801 4325 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:33.583820 4216 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:33.606268 4325 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:33.584581 4249 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:33.607191 4325 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.608151 4249 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.588573 2287 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
W20260927 23:31:33.609516 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1b945905616b4989853b0a92bfbd6b98 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:33.592517 4286 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.593441 4310 raft_consensus.cc:3032] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:33.600807 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1583) took 4207 ms. Trace:
W20260927 23:31:33.612191 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 131233us
W20260927 23:31:33.613597 1703 leader_election.cc:334] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Timed out: RequestConsensusVote RPC to 127.0.99.4:45509 timed out after 2.060s (SENT)
W20260927 23:31:33.622278 1628 connection.cc:232] RPC call timeout handler was delayed by -1.2834243539999761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260927 23:31:33.623065 1628 leader_election.cc:334] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.807s (SENT)
W20260927 23:31:33.624076 1628 leader_election.cc:334] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.128s (SENT)
I20260927 23:31:33.624511 1628 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:33.629706 4205 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:33.631338 1628 leader_election.cc:334] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.191s (SENT)
W20260927 23:31:33.632375 1628 leader_election.cc:334] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.649s (SENT)
W20260927 23:31:33.633273 1628 connection.cc:232] RPC call timeout handler was delayed by -1.1547243539999954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260927 23:31:33.633870 1628 leader_election.cc:334] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.079s (SENT)
I20260927 23:31:33.611459 1654 rpcz_store.cc:276] 0927 23:31:29.393062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:29.404493 (+ 11431us) service_pool.cc:227] Handling call
0927 23:31:29.530850 (+126357us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:30.749929 (+1219079us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:33.600534 (+2850605us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54004466d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:33.600709 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2801912576}
I20260927 23:31:33.652980 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:33.611892 4224 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:33.658111 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.646701 4310 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:33.662631 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:33.646643 4292 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:33.663830 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 58970.6us
I20260927 23:31:33.665349 1657 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:33.666213 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:33.666672 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:33.667407 1729 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:33.667019 1733 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:33.693377 1700 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:33.700217 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:33.706648 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 72805.1us
I20260927 23:31:33.611860 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:33.719506 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.720618 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.721572 1662 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:33.724558 1628 connection.cc:232] RPC call timeout handler was delayed by -1.0505337409999811s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4853s for a response
W20260927 23:31:33.730111 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 455807us
W20260927 23:31:33.699107 4190 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.748s user 0.018s sys 0.021s
I20260927 23:31:33.664060 4224 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.735901 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.739169 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.744351 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:33.747406 1654 raft_consensus.cc:1226] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:33.704509 4209 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:33.751427 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:33.741470 4190 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.790s user 0.018s sys 0.021s
I20260927 23:31:33.675586 4292 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.743783 4262 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:33.758960 1627 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:33.759806 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 276509us
I20260927 23:31:33.775134 2258 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.761917 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.773411 1729 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:33.779733 2258 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.782158 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.766927 4216 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:33.783339 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:33.784181 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 276664us
W20260927 23:31:33.780038 4343 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
I20260927 23:31:33.809589 1627 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:33.780921 4130 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:33.756613 4348 consensus_queue.cc:1046] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.056s
I20260927 23:31:33.768963 4287 consensus_queue.cc:1046] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260927 23:31:33.827056 1628 leader_election.cc:334] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.212s (SENT)
I20260927 23:31:33.829267 1628 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:33.811005 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 195183us
I20260927 23:31:33.803457 4309 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:33.806653 4319 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:33.837852 4319 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:33.837970 4351 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:33.838560 4319 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:33.750818 4310 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:33.842758 4310 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:33.808267 4262 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:33.846524 4262 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:33.846933 4262 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:33.807338 4223 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:33.808750 4216 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.856109 4216 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.857314 4348 raft_consensus.cc:3032] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:33.860303 1662 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:33.862596 4223 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 23:31:33.858210 4348 raft_consensus.cc:729] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:33.867358 4348 consensus_queue.cc:260] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.869078 4348 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:33.836771 4344 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.827976 4336 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:33.879020 4336 raft_consensus.cc:686] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:33.880200 4336 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:33.882392 4029 consensus_queue.cc:1159] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1b945905616b4989853b0a92bfbd6b98" responder_term: 5 status { last_received { term: 5 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260927 23:31:33.886180 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 102887us
I20260927 23:31:33.887180 1552 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:33.900754 1628 leader_election.cc:334] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.713s (SENT)
I20260927 23:31:33.901423 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:33.828564 4130 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:33.842895 4287 raft_consensus.cc:3032] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:33.937899 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50832us
I20260927 23:31:33.938771 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.940676 4205 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:33.883936 4262 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:33.936741 4325 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 23:31:33.972353 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 167350us
W20260927 23:31:33.948312 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 133195us
W20260927 23:31:33.976833 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 181099us
W20260927 23:31:33.979974 1628 leader_election.cc:334] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.746s (SENT)
I20260927 23:31:33.984961 1628 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.985095 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.941881 4130 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:33.988198 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.988832 4262 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:33.989416 4262 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.989452 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:33.991374 4342 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:33.991423 4309 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:33.995048 4292 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:34.003654 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.004583 1581 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:34.006716 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.007725 1581 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:33.983777 4283 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:34.009136 4283 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.010344 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.015488 1581 raft_consensus.cc:2396] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 23:31:34.011658 4283 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.017113 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.018311 2257 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:33.937465 4287 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:34.017732 1580 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:34.020056 4350 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:34.022912 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.023169 1703 leader_election.cc:334] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509): Timed out: RequestConsensusVote RPC to 127.0.99.4:45509 timed out after 1.723s (SENT)
I20260927 23:31:34.023028 4350 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.024026 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.024572 1585 raft_consensus.cc:2362] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7. Re-sending same reply.
I20260927 23:31:34.025321 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.027755 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.023651 1583 raft_consensus.cc:2354] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 8. Current term is 9.
I20260927 23:31:34.028299 1584 raft_consensus.cc:2362] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7. Re-sending same reply.
I20260927 23:31:34.030829 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.031250 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 269419us
I20260927 23:31:34.031620 1582 raft_consensus.cc:2396] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260927 23:31:34.036454 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 56518.1us
I20260927 23:31:34.037842 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.040506 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.041760 1581 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:34.043227 1702 leader_election.cc:334] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.584s (SENT)
I20260927 23:31:34.043779 1702 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.039363 1660 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:34.048560 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:34.046018 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.050486 1583 raft_consensus.cc:2362] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:34.020480 4277 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:34.056067 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.054878 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.036257 4344 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.036616 4341 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:34.036748 4352 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:34.079030 1702 connection.cc:232] RPC call timeout handler was delayed by -3.5526878489999945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2705s for a response
W20260927 23:31:34.079602 1702 leader_election.cc:334] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.444s (SENT)
I20260927 23:31:34.038393 4190 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260927 23:31:34.062296 4310 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:34.071256 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.085341 1660 raft_consensus.cc:1226] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:34.087936 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.096829 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.063591 4362 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:34.092041 1628 leader_election.cc:334] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 4.853s (SENT)
I20260927 23:31:34.091588 1702 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.066988 4223 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:34.100606 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.104598 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.076004 4350 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:34.108476 1628 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:34.110555 1588 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:34.105266 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.033929 4319 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:34.114980 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 143725us
I20260927 23:31:34.118928 1584 raft_consensus.cc:2362] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
W20260927 23:31:34.119122 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 64186.6us
W20260927 23:31:34.118053 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 87855.3us
I20260927 23:31:34.122255 2259 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:34.090289 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:34.066138 4208 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.279s user 0.032s sys 0.025s
I20260927 23:31:34.098243 4310 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:34.131332 1554 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.132979 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.133280 1554 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.133595 1582 raft_consensus.cc:2362] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
I20260927 23:31:34.083791 4352 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.137768 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.138962 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59373.4us
W20260927 23:31:34.141749 1702 connection.cc:232] RPC call timeout handler was delayed by -3.368384913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.35600000000000004s for a response
I20260927 23:31:34.128907 4223 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:34.144785 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.145259 1702 connection.cc:232] RPC call timeout handler was delayed by -2.2075857980000251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260927 23:31:34.145537 1582 raft_consensus.cc:2379] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
W20260927 23:31:34.145682 1702 connection.cc:232] RPC call timeout handler was delayed by -1.7863857980000262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20260927 23:31:34.127257 4208 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.443s user 0.033s sys 0.025s
W20260927 23:31:34.145988 1702 connection.cc:232] RPC call timeout handler was delayed by -1.7458857980000175s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
I20260927 23:31:34.130887 4310 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:34.075623 4287 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:34.152443 1702 connection.cc:232] RPC call timeout handler was delayed by -1.6576857980000455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23340000000000002s for a response
I20260927 23:31:34.131649 4224 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:34.155169 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:34.155442 1702 connection.cc:232] RPC call timeout handler was delayed by -1.2715857980000465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2763s for a response
I20260927 23:31:34.156029 1581 raft_consensus.cc:2362] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7. Re-sending same reply.
I20260927 23:31:34.140424 4262 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:34.140048 4307 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:34.165797 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.187507 1734 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:34.167096 1735 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:34.170105 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.172168 2285 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:34.172045 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.189069 1655 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:34.173026 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.174079 1737 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:34.151113 4277 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:34.192098 1584 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:34.193536 1732 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:34.191321 4277 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:34.194555 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.194718 4277 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:34.195197 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.197227 1582 raft_consensus.cc:1261] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:34.197108 4249 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:34.146996 4292 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.120787 4362 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.135886 4283 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.057s
I20260927 23:31:34.213384 1735 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:34.216106 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.228855 1584 raft_consensus.cc:2362] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5. Re-sending same reply.
W20260927 23:31:34.218786 4368 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260927 23:31:34.219442 4367 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260927 23:31:34.226682 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 134654us
W20260927 23:31:34.226997 4360 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260927 23:31:34.231377 4208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46178 (request call id 899) took 10458 ms. Trace:
I20260927 23:31:34.205790 4374 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:34.230352 4287 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:34.259454 1588 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:34.156957 4372 consensus_queue.cc:1046] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
W20260927 23:31:34.260924 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 76533.6us
I20260927 23:31:34.261016 1582 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:34.264153 4190 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:34.267536 1655 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:34.267845 4190 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.268590 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.269843 4190 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.270138 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.270833 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.271353 4262 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:34.270561 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.271831 4262 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.272325 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.272534 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.272858 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.273646 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.273834 4262 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.274292 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.269860 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.275300 2292 raft_consensus.cc:2362] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7. Re-sending same reply.
I20260927 23:31:34.265244 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.277115 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.287032 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.289263 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.293116 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.293233 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.293902 4297 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.294299 4277 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.255033 4208 rpcz_store.cc:276] 0927 23:31:23.771206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:23.771370 (+ 164us) service_pool.cc:227] Handling call
0927 23:31:25.471829 (+1700459us) spinlock_profiling.cc:243] Waited 9.31 ms on lock 0x7b54000594a0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a04dc0 00007f9750a75bd2 00007f9750a74890 00007f9750a6dc1a 00007f97559b16d9 00007f974f09b184 00007f974f09b112 00007f974f09b041 00007f974f09afbc 00007f974f09a272
0927 23:31:25.479364 (+ 7535us) log.cc:842] Serialized 480 byte log entry
0927 23:31:26.741562 (+1262198us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b180023c7a0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974dabe4ca 00007f975719ff99 00007f974d982990 00007f974db44864 00007f974db44155 00007f97509e9320 00007f9750a4f113 00007f9750a757e2 00007f9750a6dc1a
0927 23:31:34.211698 (+7470136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":262319,"raft.queue_time_us":353460,"raft.run_cpu_time_us":1692,"raft.run_wall_time_us":179749,"spinlock_wait_cycles":40442496,"thread_start_us":930427,"threads_started":1,"wal-append.queue_time_us":1339900,"wal-append.run_cpu_time_us":3368,"wal-append.run_wall_time_us":1587363}
I20260927 23:31:34.296301 2294 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:34.297168 1470 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9c719671f35640549c1b944112b3fc03 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20260927 23:31:34.297823 4267 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.278452 4341 consensus_queue.cc:1046] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260927 23:31:34.289985 2289 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:34.298779 1660 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:34.288968 4262 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260927 23:31:34.299815 1733 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:34.302983 4372 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:34.303427 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.303524 4377 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 23:31:34.304416 4377 consensus_queue.cc:1225] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260927 23:31:34.304531 4366 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:34.306435 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.412178 1655 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 4: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:34.413695 4369 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20260927 23:31:34.318162 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.317199 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.416968 1582 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:34.380808 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.417671 1587 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
W20260927 23:31:34.374214 4360 log.cc:924] Time spent T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260927 23:31:34.339084 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.380683 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.373495 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.373323 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.307176 1702 leader_election.cc:334] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 2.705s (SENT)
I20260927 23:31:34.415231 1581 raft_consensus.cc:2362] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4. Re-sending same reply.
I20260927 23:31:34.408236 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.425984 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.315434 1554 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.423774 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:34.426697 1588 raft_consensus.cc:1261] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 23:31:34.426981 1728 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:34.288393 4364 raft_consensus.cc:3032] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:34.425048 1702 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.426417 1583 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:34.425055 1585 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:34.271929 4362 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:34.429610 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.427137 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 111805us
I20260927 23:31:34.316972 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.340026 4277 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:34.433738 4217 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:34.437153 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.424096 4319 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.440114 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.443186 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.441833 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.444051 2288 raft_consensus.cc:2379] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:34.445058 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.264115 4374 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.445878 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.455792 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.456085 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.455830 2288 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260927 23:31:34.456988 4310 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.457402 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.457959 1582 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:34.458549 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.458999 2259 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:34.459389 2288 raft_consensus.cc:2396] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260927 23:31:34.463151 4130 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.464030 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.464296 2291 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:34.464604 4385 raft_consensus.cc:2781] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:34.465261 4385 raft_consensus.cc:686] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:34.460197 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.465828 4267 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.466215 4385 consensus_queue.cc:237] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.467213 4130 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.467675 1703 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.455806 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.468995 2292 raft_consensus.cc:2379] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:34.446345 4352 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.471177 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.446203 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.472108 1579 raft_consensus.cc:2396] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 23:31:34.471930 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.472725 1581 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 5: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:34.474898 1703 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.475488 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:34.477532 1703 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.480080 1703 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.486423 2289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.444511 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.487914 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.488080 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 180941us
I20260927 23:31:34.499600 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:34.491672 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:34.502763 1661 raft_consensus.cc:3037] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 6
I20260927 23:31:34.504772 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:34.493811 1702 leader_election.cc:334] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501): Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 3.560s (SENT)
I20260927 23:31:34.429456 4364 raft_consensus.cc:729] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:34.512638 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.516439 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:34.457360 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.487707 4217 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.488996 4352 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:34.478269 4223 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:34.478302 4310 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.491087 4242 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:34.529032 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 47631.7us
I20260927 23:31:34.532714 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.531533 1728 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:34.535919 1627 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:34.540059 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.543754 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.540556 1629 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:34.546567 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:34.547758 4374 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:34.548017 1732 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:34.548313 4374 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:34.492326 4341 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260927 23:31:34.541258 1584 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:34.561237 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.525183 4350 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:34.566490 1654 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:34.567896 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 87880.3us
I20260927 23:31:34.524225 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:34.525194 4352 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:34.570785 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.490886 4383 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260927 23:31:34.577214 1728 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:34.571832 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:34.575352 1661 raft_consensus.cc:2454] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:34.579988 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.509809 4362 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260927 23:31:34.543331 4217 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:34.558548 4242 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:34.559826 4316 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:34.598719 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.579052 4307 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:34.579982 4352 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:34.584158 2290 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:34.600517 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:34.600700 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 75267.7us
I20260927 23:31:34.566651 4136 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20260927 23:31:34.591337 4209 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.006s sys 0.031s
I20260927 23:31:34.603829 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.602950 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:34.616415 4209 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.849s user 0.007s sys 0.031s
I20260927 23:31:34.616753 4209 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:34.609593 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.607936 2258 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:34.613741 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.621338 1737 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:34.621980 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.622735 1729 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:34.622992 2258 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:34.624212 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.625005 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.625710 1737 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 2.
I20260927 23:31:34.602952 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:34.626853 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.627657 1730 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:34.625944 4351 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:34.628469 4351 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:34.617678 4209 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.628844 4341 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.634976 1580 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:34.635519 1703 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.636121 1583 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:34.639470 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.643601 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.645134 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.594074 4378 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
W20260927 23:31:34.657502 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 163728us
I20260927 23:31:34.655805 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.654057 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.657817 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.628855 4351 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:34.655467 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.669492 2292 raft_consensus.cc:1226] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:34.671193 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:34.677477 1703 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.664505 4375 raft_consensus.cc:3032] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 23:31:34.663239 2287 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 521) took 2580 ms (client timeout 1989 ms). Trace:
W20260927 23:31:34.681870 2287 rpcz_store.cc:269] 0927 23:31:32.082587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:32.083863 (+ 1276us) service_pool.cc:227] Handling call
0927 23:31:34.663046 (+2579183us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400640ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:34.663148 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3432578944}
I20260927 23:31:34.680528 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.683962 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.658923 4389 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:34.676980 4325 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:34.700790 1735 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:34.704348 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.704167 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.705588 1579 raft_consensus.cc:1226] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:34.706363 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:34.680235 4375 raft_consensus.cc:729] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:34.711745 4050 raft_consensus.cc:3032] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:34.711644 4209 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:34.712380 4050 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:34.712558 1702 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.713305 4209 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:34.713461 4050 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.687994 4396 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:34.730307 4375 consensus_queue.cc:260] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:34.731482 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 140859us
W20260927 23:31:34.736310 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 134739us
I20260927 23:31:34.738534 4242 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:34.704262 4325 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.739454 4242 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.740548 4325 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.744190 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:34.711084 2290 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:34.750674 1736 raft_consensus.cc:1226] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 23:31:34.751775 1629 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:34.756878 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:34.782876 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 193672us
I20260927 23:31:34.769300 4216 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:34.748945 4299 consensus_queue.cc:1046] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260927 23:31:34.778246 4050 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:34.758054 4383 consensus_queue.cc:1046] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260927 23:31:34.801697 4216 raft_consensus.cc:686] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:34.857637 4403 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260927 23:31:34.859390 4351 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.860587 2286 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:34.861701 4351 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.867513 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:34.873128 4375 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:34.800177 4336 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260927 23:31:34.878628 4216 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.880208 1579 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:34.884666 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:34.885560 1736 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:34.881801 4398 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260927 23:31:34.894335 4407 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260927 23:31:34.839601 4316 consensus_queue.cc:1035] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260927 23:31:34.930837 4396 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:34.931917 4326 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:34.932466 4292 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:34.932994 4292 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:34.932643 4326 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:34.916787 4352 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:34.907877 4249 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260927 23:31:34.940832 4352 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:34.941028 2287 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:34.942148 4402 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 23:31:34.944484 1736 raft_consensus.cc:2454] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:34.945886 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:34.951148 2258 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:34.967301 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.968001 1582 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:34.969271 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:34.974922 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.978631 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.979434 1584 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:34.978166 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.981778 1585 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:34.984534 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.984688 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.985410 1587 raft_consensus.cc:2396] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260927 23:31:34.988538 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.989607 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.989291 1579 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:34.996073 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.998298 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.000459 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.933830 4292 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.001281 1582 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:35.003095 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.006930 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.004129 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.008147 1584 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:34.975879 4366 raft_consensus.cc:2781] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:35.009852 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.010633 1582 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:35.011914 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.013052 1587 raft_consensus.cc:2396] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 23:31:35.015908 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.019397 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.020563 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.021243 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.020254 1582 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:35.025309 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.026700 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.027837 1585 raft_consensus.cc:2379] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:35.028671 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.033164 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.035176 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.046794 1733 raft_consensus.cc:1226] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:35.046677 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:35.047345 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:34.976946 4352 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260927 23:31:35.050136 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.051573 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.052695 1584 raft_consensus.cc:2396] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260927 23:31:35.032648 4366 raft_consensus.cc:686] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:35.053251 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.989566 4336 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260927 23:31:35.054284 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.062966 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:34.989387 4318 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260927 23:31:35.061121 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:35.063042 1584 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:35.055123 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.058091 1585 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:35.065037 1587 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:35.064549 1654 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 23:31:35.066197 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.050009 4277 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:35.069974 1552 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.066924 1582 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:35.019814 4205 consensus_queue.cc:1046] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.043s
I20260927 23:31:35.072829 1702 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:35.064105 4401 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.222s user 0.001s sys 0.001s
I20260927 23:31:35.056960 4352 consensus_queue.cc:1225] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer a2a3317e8da742d9aefe4c251abd4566 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260927 23:31:35.075374 1552 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.056474 4362 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.084537 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.077698 1585 pending_rounds.cc:76] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 2
I20260927 23:31:35.038290 4216 consensus_queue.cc:1046] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260927 23:31:35.085129 1584 raft_consensus.cc:1261] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 23:31:35.091557 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.092577 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:35.085551 1585 pending_rounds.cc:97] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 23:31:35.048401 4329 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260927 23:31:35.094319 1585 raft_consensus.cc:2866] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:35.095970 1702 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.076453 4297 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.088162 4287 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.051440 4364 consensus_queue.cc:260] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:35.127313 1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48976 (request call id 245) took 20798 ms (client timeout 15000 ms). Trace:
I20260927 23:31:35.093725 4404 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:35.074144 4050 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260927 23:31:35.093390 4283 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.093933 4326 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:35.157163 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 61295.8us
I20260927 23:31:35.158003 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:35.101997 4310 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 23:31:35.164970 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 72437.9us
I20260927 23:31:35.165891 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:35.182394 2292 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:35.182804 1552 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.227238 2292 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:35.227952 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45276.1us
I20260927 23:31:35.129554 4263 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.171983 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.234483 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:35.233984 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 62069.1us
I20260927 23:31:35.235987 1702 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.242466 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.230192 2292 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.183104 4310 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.250974 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.177882 4396 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:35.188336 4364 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:35.233707 4326 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:35.127751 1486 rpcz_store.cc:269] 0927 23:31:14.170567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:14.171642 (+ 1075us) service_pool.cc:227] Handling call
0927 23:31:34.968359 (+20796717us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:27.590827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:27.642759 (+ 51932us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:27.642982 (+ 223us) write_op.cc:435] Acquired schema lock
0927 23:31:27.643133 (+ 151us) tablet.cc:660] Decoding operations
0927 23:31:27.730040 (+ 86907us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:27.741872 (+ 11832us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:27.745070 (+ 3198us) tablet.cc:683] Acquiring locks for 10 operations
0927 23:31:27.884077 (+139007us) tablet.cc:699] Row locks acquired
0927 23:31:27.897494 (+ 13417us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b18002dcfa0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db44155 00007f975447d714 00007f9754441b28 00007f9756af61f1
0927 23:31:27.898671 (+ 1177us) write_op.cc:260] PREPARE: finished
0927 23:31:27.898937 (+ 266us) write_op.cc:270] Start()
0927 23:31:27.899230 (+ 293us) write_op.cc:276] Timestamp: P: 1790551887898847 usec, L: 0
0927 23:31:27.899290 (+ 60us) op_driver.cc:331] REPLICATION: starting
0927 23:31:27.900110 (+ 820us) log.cc:842] Serialized 3930 byte log entry
0927 23:31:28.314012 (+413902us) op_driver.cc:444] REPLICATION: finished
0927 23:31:29.786554 (+1472542us) write_op.cc:301] APPLY: starting
0927 23:31:29.802988 (+ 16434us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:29.836692 (+ 33704us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:29.836729 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:30.014384 (+177655us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:30.014791 (+ 407us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0927 23:31:30.014846 (+ 55us) write_op.cc:312] APPLY: finished
0927 23:31:30.015267 (+ 421us) log.cc:842] Serialized 77 byte log entry
0927 23:31:30.120529 (+105262us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:30.128851 (+ 8322us) write_op.cc:454] Released schema lock
0927 23:31:30.139579 (+ 10728us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:30.166636 (+ 27057us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:30.169790 (+ 3154us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6017,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":1322084,"apply.run_cpu_time_us":20891,"apply.run_wall_time_us":431637,"num_ops":10,"prepare.queue_time_us":13220,"prepare.run_cpu_time_us":20540,"prepare.run_wall_time_us":432430,"raft.queue_time_us":242228,"raft.run_cpu_time_us":23658,"raft.run_wall_time_us":435842,"replication_time_us":408543,"spinlock_wait_cycles":225091840,"thread_start_us":534513,"threads_started":5,"wal-append.queue_time_us":218736,"wal-append.run_cpu_time_us":37757,"wal-append.run_wall_time_us":2209379}]]}
I20260927 23:31:35.233034 4263 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:35.241436 4362 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:35.241848 4389 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.167186 4348 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.281463 2258 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.280805 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.178169 4352 consensus_queue.cc:1035] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.096s
W20260927 23:31:35.289767 4415 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260927 23:31:35.290839 2287 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260927 23:31:35.124799 4366 consensus_queue.cc:237] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.169157 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be2a211b7754af89c38527623e5d967 with delete_type TABLET_DATA_TOMBSTONED (TS 1b945905616b4989853b0a92bfbd6b98 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:46178
I20260927 23:31:35.275168 4297 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.266422 4267 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 23:31:35.291244 2287 pending_rounds.cc:76] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting all ops after (but not including) 6
I20260927 23:31:35.264495 4310 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:35.277088 4209 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.295949 4224 raft_consensus.cc:2741] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260927 23:31:35.299448 4404 consensus_queue.cc:1046] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 23:31:35.300071 2286 raft_consensus.cc:1226] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:35.300797 4416 log.cc:924] Time spent T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.135s user 0.000s sys 0.000s
I20260927 23:31:35.448699 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1451) took 1129 ms. Trace:
W20260927 23:31:35.301672 4413 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20260927 23:31:35.265604 1552 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.295660 4326 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:35.449926 4326 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:35.452129 4326 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:35.312023 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:35.453059 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 141117us
I20260927 23:31:35.453809 1702 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:35.316179 4267 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:35.454764 4267 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:35.454890 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.312630 4389 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:35.287631 4263 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:35.292450 4348 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:35.292670 4362 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:35.449105 1586 rpcz_store.cc:276] 0927 23:31:34.319100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:34.319993 (+ 893us) service_pool.cc:227] Handling call
0927 23:31:34.381265 (+ 61272us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:34.381682 (+ 417us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:34.381733 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:34.948223 (+566490us) spinlock_profiling.cc:243] Waited 8.11 ms on lock 0x7b2c00029428. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974f452e11 00007f974f462442 00007f9750aaac11 00007f9750a6f58a 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772
0927 23:31:34.948658 (+ 435us) log.cc:842] Serialized 26 byte log entry
0927 23:31:35.058697 (+110039us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:35.273048 (+214351us) log.cc:842] Serialized 14 byte log entry
0927 23:31:35.273311 (+ 263us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:35.273378 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:35.448443 (+175065us) raft_consensus.cc:1671] finished
0927 23:31:35.448538 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:35.448649 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6027,"prepare.queue_time_us":25343,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":182,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":22527,"spinlock_wait_cycles":18646784,"thread_start_us":304468,"threads_started":2,"wal-append.queue_time_us":149219}
W20260927 23:31:35.456825 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 191318us
I20260927 23:31:35.481333 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.317960 2287 pending_rounds.cc:97] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting uncommitted NO_OP operation due to leader change: 8.7
W20260927 23:31:35.531260 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.018s sys 0.025s
W20260927 23:31:35.531721 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.018s sys 0.025s
I20260927 23:31:35.318334 4310 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:35.551077 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 589892us
W20260927 23:31:35.559691 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 78466.5us
I20260927 23:31:35.437186 4209 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:35.560899 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.566579 4427 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:35.568840 4424 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260927 23:31:35.532528 2287 raft_consensus.cc:2866] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:35.480645 4389 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:35.532102 4287 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.509850 4362 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:35.481415 4348 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:35.580071 1552 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.532028 4297 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.507982 4396 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:35.578785 4209 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:35.584378 4326 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:35.584049 4325 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:35.605710 4396 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.612259 4277 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:35.614286 4427 consensus_queue.cc:1046] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260927 23:31:35.586565 4389 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:35.637531 1552 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.612011 4365 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:35.609259 4423 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.641682 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0be2a211b7754af89c38527623e5d967"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:35.642581 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:35.649717 1581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1459) took 1194 ms (client timeout 1560 ms). Trace:
W20260927 23:31:35.676821 1581 rpcz_store.cc:269] 0927 23:31:34.455137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:34.455538 (+ 401us) service_pool.cc:227] Handling call
0927 23:31:35.649542 (+1194004us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b540013f6d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:35.649666 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37173120}
I20260927 23:31:35.641211 4326 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.654430 4303 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:35.678606 4303 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:35.679423 4303 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.680621 4303 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:35.681996 4029 raft_consensus.cc:3032] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:35.682418 4029 raft_consensus.cc:729] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:35.682947 4427 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.654824 4267 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.646914 4365 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.656208 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:35.649446 4432 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
W20260927 23:31:35.694413 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 239564us
I20260927 23:31:35.627203 4277 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.683383 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.628572 4325 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:35.695232 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.700176 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:35.700629 1588 raft_consensus.cc:1226] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:35.702104 1733 raft_consensus.cc:1261] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:35.700496 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.705269 1735 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:35.703579 1702 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:35.711516 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 159112us
I20260927 23:31:35.712172 1702 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:35.683244 4029 consensus_queue.cc:260] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.718453 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.721724 1552 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.703253 4326 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:35.708952 4427 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:35.707686 4287 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.717480 4396 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.766940 4325 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:35.767197 4429 log.cc:924] Time spent T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260927 23:31:35.768556 4267 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:35.769270 4433 tablet_replica.cc:330] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: stopping tablet replica
I20260927 23:31:35.769773 4420 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:35.771998 4389 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:35.770795 4362 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:35.773680 4329 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:35.816493 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 146047us
I20260927 23:31:35.774161 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:35.762380 4326 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.779229 4365 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:35.706202 4438 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 23:31:35.769663 4029 raft_consensus.cc:3037] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:35.837860 1733 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:35.781967 4441 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:35.753633 4297 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:35.818960 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:35.843019 1735 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:35.821051 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:35.803507 4396 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.760285 4287 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:35.849004 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 136672us
I20260927 23:31:35.832472 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:35.849912 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:35.867506 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 44186.5us
W20260927 23:31:35.872326 1702 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.535s (SENT) [suppressed 54 similar messages]
I20260927 23:31:35.852988 4287 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:35.873966 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 152056us
I20260927 23:31:35.859925 4283 raft_consensus.cc:3032] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:35.861419 4362 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:35.876835 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:35.879648 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56586.6us
W20260927 23:31:35.880518 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 116856us
W20260927 23:31:35.881510 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1b945905616b4989853b0a92bfbd6b98 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:35.884598 1735 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:35.885979 4277 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:35.893524 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1465) took 1358 ms. Trace:
I20260927 23:31:35.896430 1583 rpcz_store.cc:276] 0927 23:31:34.534822 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:34.534974 (+ 152us) service_pool.cc:227] Handling call
0927 23:31:34.635717 (+100743us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:34.636551 (+ 834us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:35.893350 (+1256799us) spinlock_profiling.cc:243] Waited 86 ms on lock 0x7b54003093d0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:35.893472 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197738368}
I20260927 23:31:35.851609 4420 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:35.859709 4310 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.876710 4283 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:35.926892 1552 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.908986 4336 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:35.921561 1661 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:35.925522 2292 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:35.928112 1552 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.930521 4267 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:35.896066 4364 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:35.939054 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 94628.1us
I20260927 23:31:35.939445 4389 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:35.940296 4418 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:35.964598 4283 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.965869 4283 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:35.966403 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:35.967612 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:35.942584 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 52154.9us
W20260927 23:31:35.966442 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 116617us
I20260927 23:31:35.945174 4292 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:35.945899 4443 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:35.951347 4263 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:35.953562 4409 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 23:31:35.935002 4420 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:35.848255 4329 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.961668 4325 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:35.962214 4364 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:35.957026 4444 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:35.957373 4336 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:35.972401 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64994.6us
I20260927 23:31:35.973026 1582 raft_consensus.cc:1226] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
W20260927 23:31:35.974234 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 46205us
I20260927 23:31:35.996676 1586 raft_consensus.cc:595] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.997093 1586 raft_consensus.cc:603] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:36.000510 1580 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:35.985484 4242 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:35.973455 4389 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:35.992012 1628 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:35.993158 2258 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:36.002933 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:36.006770 1728 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:31:36.007063 4050 consensus_queue.cc:1046] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:36.007912 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:35.967043 1662 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:36.009979 4336 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.011240 4336 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:36.011570 4364 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.012444 4325 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.003062 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.012913 4325 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.004138 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.007793 4418 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 23:31:36.002200 4444 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.014889 4325 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.006160 4242 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.016599 4242 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.016861 4444 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.021010 4301 raft_consensus.cc:3032] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:36.020889 4221 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:36.021438 4301 raft_consensus.cc:729] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:36.022476 3922 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:36.022327 4301 consensus_queue.cc:260] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.023572 4329 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.023864 4441 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:36.023950 4301 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:36.024282 4441 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:36.025074 4441 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.028229 4397 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.028780 4397 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.030267 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:36.002521 4326 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260927 23:31:36.031170 4326 consensus_queue.cc:1225] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260927 23:31:36.031754 4326 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:36.032060 4326 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:36.032774 4326 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.033864 4326 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:36.034636 4349 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:36.034461 4389 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.036800 4389 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.036864 4050 raft_consensus.cc:3032] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 23:31:36.037294 4050 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:36.037694 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.038072 4050 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.023453 4419 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:36.039357 4419 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:35.988580 4450 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:36.040328 4310 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.040486 4420 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.041023 4420 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.042069 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:36.042910 4420 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:36.043833 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51891.7us
I20260927 23:31:36.044100 4319 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:36.044469 4319 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:36.044613 1628 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.044840 4319 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.045743 1628 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.046458 3776 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.047001 3776 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:36.047541 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.048125 1588 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:36.048270 4397 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.031236 2286 raft_consensus.cc:2396] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260927 23:31:36.040145 4419 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.049832 4419 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:36.051244 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 58148.9us
I20260927 23:31:36.051452 1482 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:36.055552 4263 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.056440 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.056516 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.057009 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:36.057139 1729 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:36.057222 1734 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:36.038947 4209 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.059391 1737 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:36.059473 4209 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.062934 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.065317 1728 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:36.005290 1702 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.049127 2292 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260927 23:31:36.067200 2292 pending_rounds.cc:76] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting all ops after (but not including) 6
I20260927 23:31:36.053561 1482 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 5, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:36.069718 2288 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:36.068112 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 62912us
I20260927 23:31:36.071650 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.071870 1628 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:36.076007 2292 pending_rounds.cc:97] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260927 23:31:36.076259 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:36.077214 2292 raft_consensus.cc:2866] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:36.078507 1731 raft_consensus.cc:1226] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 2. Current term is 3. Ops: []
I20260927 23:31:36.076491 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.080255 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.075369 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.082353 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.085191 1628 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.083701 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.080739 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.153618 1736 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:36.124660 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.119956 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.122900 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:36.123507 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:36.159046 1588 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:36.130625 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.089293 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:36.160770 1660 raft_consensus.cc:2396] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260927 23:31:36.161128 2259 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:36.163038 1551 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.083590 4209 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.106729 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.137629 4441 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:36.083344 4352 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:36.120584 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:36.222100 1582 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:36.118429 4449 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:36.148483 4385 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260927 23:31:36.143476 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
W20260927 23:31:36.190074 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 83467.7us
W20260927 23:31:36.142791 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 62640.6us
I20260927 23:31:36.228605 1586 raft_consensus.cc:595] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:36.162591 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 73371.2us
I20260927 23:31:36.091195 1482 catalog_manager.cc:5495] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 3 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:36.149605 4438 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:36.223819 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 112691us
I20260927 23:31:36.118858 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:36.149489 4433 raft_consensus.cc:2229] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 23:31:36.130482 4263 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.126508 4029 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:36.150627 4363 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.087690 4283 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.128705 4348 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.120249 3922 consensus_queue.cc:1046] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.168s
I20260927 23:31:36.083770 4420 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.251312 2258 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.211287 4398 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:36.252660 1628 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.299173 1588 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:36.229274 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.166517 4422 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 23:31:36.229235 1586 raft_consensus.cc:603] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:36.230568 4307 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.190452 4397 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:36.166447 4341 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:36.069340 4389 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.134975 4050 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 2 FOLLOWER]: Advancing to term 3
W20260927 23:31:36.277036 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 52940.6us
I20260927 23:31:36.229367 1702 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.272889 1732 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260927 23:31:36.119395 2287 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:36.277143 1582 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:36.083374 4208 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:36.148331 4242 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.155889 4287 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:36.301412 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50235.9us
I20260927 23:31:36.221925 4352 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.126142 4452 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 23:31:36.305128 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 76006.9us
I20260927 23:31:36.327327 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:36.327839 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:36.328302 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 99013.4us
I20260927 23:31:36.301019 4263 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.288559 4376 raft_consensus.cc:3032] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:36.309692 1735 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:36.273748 4384 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.302635 1482 catalog_manager.cc:5495] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
W20260927 23:31:36.314479 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 61896us
I20260927 23:31:36.304335 4130 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:36.334646 1734 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 23:31:36.302461 4398 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.304421 4406 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:36.323619 4242 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.306854 4029 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.190668 4366 consensus_queue.cc:1046] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260927 23:31:36.329475 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.329514 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.311178 4341 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.331769 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:36.333009 1733 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:36.333319 1582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:36.320574 4301 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.301537 4283 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.332484 4437 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260927 23:31:36.333331 4376 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:36.309397 4307 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.341022 4347 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.334815 4130 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.341317 4316 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:36.326524 4352 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:36.303344 4420 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.346565 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:36.350863 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.348531 1482 catalog_manager.cc:5495] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 6, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 6 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:36.331769 4438 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:36.334697 4397 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:31:36.339607 4398 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:36.417843 1582 raft_consensus.cc:595] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.364053 4205 raft_consensus.cc:3032] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:36.419754 4347 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.421959 4432 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 23:31:36.423470 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 52554us
I20260927 23:31:36.344976 4287 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.301836 4363 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:36.427670 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 76868.7us
I20260927 23:31:36.424468 4377 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.004s
I20260927 23:31:36.355019 4348 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.428758 4396 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:36.429395 4283 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.337921 4384 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.513372 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.517069 2286 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
W20260927 23:31:36.431156 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 57992.8us
I20260927 23:31:36.520417 1629 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.431636 4208 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:36.523473 4208 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:36.433682 4136 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.358295 4263 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.434413 4420 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.356161 4319 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.524691 4136 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.435320 1482 catalog_manager.cc:5495] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 5 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:36.436522 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.437407 1660 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:36.438617 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.438822 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.439472 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.526258 4319 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.539847 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.593521 1657 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:36.446008 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.423117 4130 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Advancing to term 10
W20260927 23:31:36.427670 4459 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.271s user 0.004s sys 0.000s
I20260927 23:31:36.352494 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:36.603614 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 251191us
I20260927 23:31:36.604355 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.604583 4130 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.605903 2288 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260927 23:31:36.444074 1582 raft_consensus.cc:603] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:36.606920 4283 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:36.607424 4307 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.607558 4283 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.425668 4433 raft_consensus.cc:2258] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 23:31:36.354715 4326 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.422925 4449 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.610661 4449 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:36.444959 4466 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.235s user 0.001s sys 0.000s
I20260927 23:31:36.446478 4301 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:36.356597 4249 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.463680 1702 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.612444 4287 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:36.424801 4364 raft_consensus.cc:3032] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:36.515830 4384 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.526655 4136 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.613013 4364 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:36.612879 4287 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.613734 4287 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.613922 4364 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.613040 1658 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:36.615149 4364 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:36.429978 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 77290.1us
I20260927 23:31:36.616500 4326 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.616950 4326 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.618975 4326 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:36.621050 4287 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:36.622614 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 159039us
I20260927 23:31:36.623204 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.623263 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.623934 1581 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:36.624168 1587 raft_consensus.cc:2379] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 9: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:36.625321 1584 raft_consensus.cc:1261] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 23:31:36.626713 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.626861 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.627431 4136 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:36.627521 1585 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:36.627461 4396 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:36.628433 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:36.629030 4422 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.627992 4136 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.545847 4303 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:36.547271 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.550118 1482 catalog_manager.cc:5495] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 5 to 6, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 6 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:36.551376 4352 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.635797 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.643818 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:36.556267 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.659193 1660 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:36.645044 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.355412 4209 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:36.533835 4347 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.565675 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.669348 1734 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:36.661198 1656 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:36.667790 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:36.670677 2288 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:36.668219 1583 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260927 23:31:36.532330 4363 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.570763 1703 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:36.673470 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 102780us
I20260927 23:31:36.572897 4263 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:36.575047 4463 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260927 23:31:36.553936 4406 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:36.586804 1659 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:36.538568 3922 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:36.685316 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:36.365785 4341 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:36.610977 1582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:36.612142 4249 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:36.541790 4419 raft_consensus.cc:659] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20260927 23:31:36.612711 4449 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.448586 4397 raft_consensus.cc:686] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:36.453212 4205 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:36.629370 1702 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:36.636884 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:36.719838 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 83110.5us
I20260927 23:31:36.660722 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:36.632494 1585 pending_rounds.cc:76] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 3
I20260927 23:31:36.630242 4437 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:36.687286 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:36.687167 1482 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 5 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:36.689440 1587 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260927 23:31:36.686149 1731 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:36.728058 1663 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:36.631538 4136 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.699988 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.682157 4450 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:36.739023 1482 catalog_manager.cc:5495] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 5, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:36.687037 1582 raft_consensus.cc:595] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:36.667912 4283 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:36.665014 4438 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.668691 4209 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.720990 1585 pending_rounds.cc:97] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.4
W20260927 23:31:36.721110 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 60516.3us
W20260927 23:31:36.724925 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 95612.5us
I20260927 23:31:36.685037 4406 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:36.543180 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.758140 1585 raft_consensus.cc:2866] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 23:31:36.732739 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 45527.3us
I20260927 23:31:36.734681 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.634477 4303 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.668856 4301 consensus_queue.cc:1046] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260927 23:31:36.730805 1581 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:36.791206 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.629540 4422 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.772337 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:36.784498 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:36.787879 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:36.461030 4376 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.796339 1736 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:36.686370 4263 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.802523 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:36.803718 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:36.728562 4449 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:36.749388 4372 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:36.749092 4050 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:36.826160 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:36.832146 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 384334us
I20260927 23:31:36.832474 1659 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:36.741822 4347 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:36.738145 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:36.809278 4433 ts_tablet_manager.cc:1905] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 23:31:36.810352 4352 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.811336 4397 consensus_queue.cc:237] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.821125 4422 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.826362 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:36.700109 3922 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.755673 4283 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.772509 4209 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:36.840646 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.753778 4450 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.833280 4372 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:36.841810 1736 raft_consensus.cc:2454] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 10.
I20260927 23:31:36.848719 1587 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.795450 4287 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:36.850435 1659 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:36.900996 2257 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:36.825623 4384 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.858222 1661 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:36.824728 3776 raft_consensus.cc:3032] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 23:31:36.830992 4449 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:36.751106 4130 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:36.858134 4447 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.870968 1628 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:36.796147 4351 consensus_queue.cc:1035] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260927 23:31:36.841580 4347 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.934955 1482 catalog_manager.cc:5495] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 4, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:36.840068 4050 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:36.943405 1581 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:36.902881 4029 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:36.903544 4372 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:36.867137 4283 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:36.982842 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 112018us
I20260927 23:31:36.998749 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:36.899046 4310 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:36.799822 4205 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:36.941180 4352 raft_consensus.cc:2781] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:36.937741 4449 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:36.776371 4438 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.818305 4398 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:36.936967 3776 raft_consensus.cc:729] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:36.980417 4347 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:37.016707 4440 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:36.987876 4050 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:36.985584 4452 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:37.015341 1482 catalog_manager.cc:5495] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:36.941406 4384 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.006632 4249 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.057703 1482 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 2 to 3, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 3 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:36.974701 4447 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.044528 4205 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.078194 3922 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.044271 4352 raft_consensus.cc:686] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:37.081489 4341 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.061784 4444 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:37.088729 4447 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.088688 1587 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:37.024994 4440 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:37.098819 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.079452 4384 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.084236 4406 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.012089 4310 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.109266 1482 catalog_manager.cc:5495] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:37.122763 1659 raft_consensus.cc:2379] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:37.125720 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 292310us
I20260927 23:31:37.102795 4422 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:37.105261 4287 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
W20260927 23:31:37.163010 4471 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260927 23:31:37.177330 1660 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:37.176615 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 203475us
I20260927 23:31:37.124905 4440 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.203274 2258 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:37.179219 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.146785 4263 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.151475 4303 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.214278 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:37.215174 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:37.221266 1736 raft_consensus.cc:3037] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:37.225165 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:37.123013 4444 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.230522 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:37.230865 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:37.225867 1730 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:37.231525 2292 raft_consensus.cc:2354] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:37.199829 4437 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:37.211963 1482 catalog_manager.cc:5495] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 1 to 3, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } }
I20260927 23:31:37.221758 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.233772 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.234140 1733 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:37.175148 4341 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.107895 3776 consensus_queue.cc:260] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.235678 1731 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:37.235662 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:37.244376 1629 leader_election.cc:398] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:37.256409 1736 raft_consensus.cc:2454] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:37.271099 1629 leader_election.cc:401] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:37.222532 4352 consensus_queue.cc:237] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:37.258643 4474 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.188s user 0.000s sys 0.003s
W20260927 23:31:37.289047 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 53494us
I20260927 23:31:37.257941 1730 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:37.259927 4310 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.262372 4450 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.214303 4384 raft_consensus.cc:2781] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 23:31:37.243289 4398 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:37.274530 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.281147 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.300040 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.300844 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.301928 1733 raft_consensus.cc:2396] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:37.221553 4263 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:37.290613 2258 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:37.289907 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.292074 4287 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:37.292505 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.296861 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.204350 4422 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.300203 1662 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:37.275590 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.316602 1659 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:37.263911 4477 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.289290 4205 raft_consensus.cc:3037] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 8
I20260927 23:31:37.304255 1482 catalog_manager.cc:5495] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 reported cstate change: term changed from 4 to 5, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 5 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:37.319010 4485 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:37.298121 4382 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:37.320482 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.324667 4496 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:37.335997 4050 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.337442 4287 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:37.394073 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 174584us
I20260927 23:31:37.338073 4389 consensus_queue.cc:1046] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260927 23:31:37.339612 4465 log.cc:924] Time spent T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.764s user 0.003s sys 0.006s
I20260927 23:31:37.340804 4396 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.341017 4422 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.343988 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.427670 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.347229 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.429177 1580 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:37.430256 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.430963 1580 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:37.345822 4438 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.365970 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:37.432140 1581 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.426705 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.366698 4444 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.369714 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:37.371151 4406 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.355134 2286 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 23:31:37.374125 4283 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.353334 4485 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:37.356233 4382 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:37.313500 4384 raft_consensus.cc:686] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:37.380326 4496 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:37.381050 4326 raft_consensus.cc:979] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:37.382038 4376 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.490419 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:37.491786 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.492386 1585 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:37.493302 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.493969 1585 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:37.488669 4440 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.353353 4477 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.387169 4347 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.396888 1554 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:37.374030 3776 raft_consensus.cc:3037] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:37.434660 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:37.498813 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 100640us
W20260927 23:31:37.499598 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 102810us
I20260927 23:31:37.438198 1587 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.442798 4303 consensus_queue.cc:1046] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:37.388631 4372 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:37.503609 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 196801us
I20260927 23:31:37.455480 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.504480 1661 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:37.457566 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:37.445290 4485 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:37.439352 4486 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260927 23:31:37.466539 4382 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:37.396255 4483 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.478135 4283 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.478747 1582 raft_consensus.cc:2354] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:37.479084 4438 raft_consensus.cc:3032] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:37.531172 4438 raft_consensus.cc:729] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:37.532243 4440 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.532001 4438 consensus_queue.cc:260] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.532747 4483 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.478875 4384 consensus_queue.cc:237] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.535041 4483 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:37.480011 4050 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.535495 4205 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:37.425026 4400 consensus_queue.cc:1046] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260927 23:31:37.433032 3922 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260927 23:31:37.536824 3922 raft_consensus.cc:3032] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:37.536963 4400 raft_consensus.cc:3032] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:37.537220 3922 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:37.537464 4400 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:37.538249 3922 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.538380 4400 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.539533 3922 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.539848 4400 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:37.483480 4406 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:37.540966 4406 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.484062 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:37.543149 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:37.543989 4347 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:37.544227 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:37.544760 1586 raft_consensus.cc:595] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:37.545058 1586 raft_consensus.cc:603] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:37.545599 4372 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:37.525416 4422 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:37.313005 4263 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.546482 2287 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:37.546514 4422 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:37.547904 4422 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:37.496253 1581 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:37.506392 1580 raft_consensus.cc:2379] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:37.508011 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.508986 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:37.552862 1736 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:37.509898 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.555845 1660 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:37.512400 1552 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.517625 1587 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:37.496944 4496 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:37.560578 1583 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:37.525266 1662 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:37.526877 2293 raft_consensus.cc:2354] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 8. Current term is 9.
W20260927 23:31:37.499792 4209 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.562s user 0.010s sys 0.002s
I20260927 23:31:37.613214 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.613744 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:31:37.528875 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:37.529583 1628 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:37.614955 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 85480.9us
I20260927 23:31:37.615787 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.617125 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:37.533188 4438 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.481446 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 430) took 1483 ms. Trace:
I20260927 23:31:37.438380 4396 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.559875 1580 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:37.569362 4263 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:37.585793 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 73511.2us
I20260927 23:31:37.618047 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.625825 2258 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:37.629832 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:37.590570 1660 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:37.630663 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:37.626183 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.632385 1552 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.633662 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.623910 1580 pending_rounds.cc:76] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 2
I20260927 23:31:37.634718 1580 pending_rounds.cc:97] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260927 23:31:37.635006 1580 raft_consensus.cc:2866] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:37.603338 4382 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.611639 1656 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:37.616019 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:37.639708 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:37.445603 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:37.618618 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:37.618052 1579 rpcz_store.cc:276] 0927 23:31:35.997851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:35.998836 (+ 985us) service_pool.cc:227] Handling call
0927 23:31:35.999243 (+ 407us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:35.999670 (+ 427us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 23:31:36.060626 (+ 60956us) log.cc:842] Serialized 14 byte log entry
0927 23:31:36.337465 (+276839us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:36.344656 (+ 7191us) log.cc:842] Serialized 26 byte log entry
0927 23:31:36.344783 (+ 127us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:36.344820 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:36.344896 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:37.339044 (+994148us) spinlock_profiling.cc:243] Waited 88.8 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:37.481136 (+142092us) raft_consensus.cc:1671] finished
0927 23:31:37.481266 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:37.481391 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39727,"prepare.queue_time_us":61191,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":5465,"raft.queue_time_us":10355,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":12377,"spinlock_wait_cycles":204134272,"thread_start_us":268043,"threads_started":1,"wal-append.queue_time_us":388059}
I20260927 23:31:37.642727 1659 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 9
I20260927 23:31:37.594038 4422 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.644909 1736 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:37.549592 4267 raft_consensus.cc:1067] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:37.628953 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:37.655915 1551 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:37.438973 4437 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.707793 1582 raft_consensus.cc:1226] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:37.709806 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:37.710976 1628 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:37.617969 4450 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:37.713430 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:37.715659 2258 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:37.747084 4385 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:37.716615 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:37.750622 1702 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:37.640074 4441 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:37.656831 4205 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:37.654983 4377 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:37.825870 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:37.707278 4263 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:37.753935 4449 consensus_meta.cc:287] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.689s user 0.008s sys 0.001s
I20260927 23:31:37.654245 1581 raft_consensus.cc:1261] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260927 23:31:38.026207 1581 pending_rounds.cc:76] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 3
I20260927 23:31:38.026818 1581 pending_rounds.cc:97] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260927 23:31:38.027159 1581 raft_consensus.cc:2866] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 23:31:38.021684 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 384655us
I20260927 23:31:38.030222 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:38.022912 1584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.005s sys 0.009s
I20260927 23:31:37.791551 1659 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:37.876914 1662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:46188
I20260927 23:31:38.037451 4524 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.037575 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1581) took 1151 ms. Trace:
I20260927 23:31:38.038090 1580 rpcz_store.cc:276] 0927 23:31:36.885733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.623189 (+737456us) service_pool.cc:227] Handling call
0927 23:31:37.623558 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:37.635557 (+ 11999us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:37.635587 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:37.636073 (+ 486us) log.cc:842] Serialized 26 byte log entry
0927 23:31:38.024503 (+388430us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:38.025054 (+ 551us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.036959 (+ 11905us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.037097 (+ 138us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:38.037178 (+ 81us) raft_consensus.cc:1671] finished
0927 23:31:38.037188 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.037399 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":293472,"threads_started":2,"wal-append.queue_time_us":315708}
I20260927 23:31:38.039407 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1484) took 1296 ms. Trace:
I20260927 23:31:38.039987 1586 rpcz_store.cc:276] 0927 23:31:36.742443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.545696 (+803253us) service_pool.cc:227] Handling call
0927 23:31:37.545990 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:37.546173 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 23:31:37.561308 (+ 15135us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.038480 (+477172us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:38.038605 (+ 125us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:38.038775 (+ 170us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.038925 (+ 150us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.039171 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79259,"prepare.queue_time_us":66826,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":16721,"thread_start_us":476324,"threads_started":2,"wal-append.queue_time_us":404895}
W20260927 23:31:37.924402 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 49228.3us
I20260927 23:31:37.654678 4050 raft_consensus.cc:2781] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:37.835042 4205 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:37.620618 4209 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.147341 4209 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:37.660946 4292 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:37.655174 4303 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:38.163867 4437 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.164517 4303 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:38.165203 4303 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.165005 4437 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.654866 4326 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:38.168051 4437 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.169268 4437 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.834661 4377 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.170101 4437 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.828835 4397 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
W20260927 23:31:38.177096 4509 log.cc:924] Time spent T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 23:31:37.947948 4406 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:37.822750 4441 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:38.142830 4050 raft_consensus.cc:686] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:38.188443 4441 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:37.838724 4449 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.819372 4450 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.917459 4468 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.189543 4050 consensus_queue.cc:237] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.191123 4468 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.192266 4449 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.193356 1730 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:37.940290 4475 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.928498 4477 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:37.709403 4483 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.827533 4485 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:37.714859 4486 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:37.750031 4495 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:37.822397 4496 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:38.223482 1654 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.609s user 0.014s sys 0.000s
I20260927 23:31:38.224036 1654 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:38.224459 2291 raft_consensus.cc:2379] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 7: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:37.834293 4497 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:37.952371 4498 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.194391 4475 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.246796 4497 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.246816 4498 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.247759 4497 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.248425 4475 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.249091 4475 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:37.992779 4518 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.249372 4498 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.250190 4518 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.251461 4301 raft_consensus.cc:3032] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:38.251995 4301 raft_consensus.cc:729] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:38.252339 4518 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:38.061676 1584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.006s sys 0.009s
I20260927 23:31:38.253813 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1472) took 1586 ms. Trace:
I20260927 23:31:38.253312 4301 consensus_queue.cc:260] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.254139 1584 rpcz_store.cc:276] 0927 23:31:36.667700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:36.668373 (+ 673us) service_pool.cc:227] Handling call
0927 23:31:36.668679 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:36.669003 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:36.669031 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:36.684945 (+ 15914us) log.cc:842] Serialized 26 byte log entry
0927 23:31:37.115995 (+431050us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:37.297844 (+181849us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.253366 (+955522us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.253496 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:38.253567 (+ 71us) raft_consensus.cc:1671] finished
0927 23:31:38.253578 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.253745 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":199232,"prepare.queue_time_us":200503,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":208050,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":13988,"thread_start_us":1325646,"threads_started":3,"wal-append.queue_time_us":549411}
I20260927 23:31:38.255779 1629 leader_election.cc:398] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 8. Current term is 9.
I20260927 23:31:38.073027 4524 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.258021 4524 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:38.103484 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 73614.8us
I20260927 23:31:38.112910 4263 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.203131 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.113765 4440 raft_consensus.cc:2741] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260927 23:31:38.116253 4389 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.122431 4382 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.123268 4396 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:38.126770 4376 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.641s user 0.009s sys 0.000s
I20260927 23:31:38.130999 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:38.139199 4433 ts_tablet_manager.cc:1918] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 23:31:38.141775 4136 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:37.968454 4422 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.143841 4513 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:37.932019 4287 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.149261 4526 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:38.149770 4130 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:38.152320 4283 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:38.293787 4130 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.293466 4287 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.153818 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1488) took 1409 ms. Trace:
W20260927 23:31:38.171195 4514 log.cc:924] Time spent T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260927 23:31:38.178004 4437 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.113766 4221 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.187027 4377 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.186820 4205 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.304108 2292 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 9: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:38.305024 4263 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.297411 4531 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.309494 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36962 (request call id 358) took 1020 ms. Trace:
I20260927 23:31:38.309608 4221 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:38.197487 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.047s sys 0.166s
I20260927 23:31:38.194957 2257 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:38.195820 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.195359 4397 raft_consensus.cc:686] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:38.234472 1731 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:38.193615 4468 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.327710 1731 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:38.236917 1733 raft_consensus.cc:1226] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 9. Current term is 10. Ops: [6.6-9.7]
I20260927 23:31:38.202785 4495 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:38.238756 4438 consensus_meta.cc:287] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.621s user 0.004s sys 0.011s
I20260927 23:31:38.195091 4326 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.243577 4209 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.199368 4486 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.244803 4449 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.195461 4483 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:37.928699 4500 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:38.113312 1662 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:38.197672 4485 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.198760 2287 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 488) took 1552 ms. Trace:
I20260927 23:31:38.363639 1662 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:38.363996 2287 rpcz_store.cc:276] 0927 23:31:36.646151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:36.657403 (+ 11252us) service_pool.cc:227] Handling call
0927 23:31:38.198302 (+1540899us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b54006dea54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:38.198528 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1606162176}
I20260927 23:31:38.364500 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:38.278679 4400 consensus_meta.cc:287] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.719s user 0.004s sys 0.013s
I20260927 23:31:38.278205 4136 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.367475 4136 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.279160 4433 log.cc:1196] T 0be2a211b7754af89c38527623e5d967 P 1b945905616b4989853b0a92bfbd6b98: Deleting WAL directory at /tmp/dist-test-taskfkrma1/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790551754024304-396-0/minicluster-data/ts-1-root/wals/0be2a211b7754af89c38527623e5d967
I20260927 23:31:38.279259 4249 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:38.279781 4475 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.280232 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:38.280930 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:38.290889 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.296330 4130 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.339143 4376 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:38.312168 1730 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:38.226784 4528 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.307866 4531 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.320078 4221 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.323817 4444 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.319846 2293 raft_consensus.cc:2354] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
W20260927 23:31:38.321193 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 126650us
I20260927 23:31:38.320465 1728 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:38.322686 4397 consensus_queue.cc:237] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.323074 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:38.315171 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.270s user 0.048s sys 0.168s
I20260927 23:31:38.291875 4513 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:38.320950 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 44293.7us
I20260927 23:31:38.317626 4389 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.295986 1587 rpcz_store.cc:276] 0927 23:31:36.743801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.561615 (+817814us) service_pool.cc:227] Handling call
0927 23:31:37.561997 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:37.609816 (+ 47819us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:37.609860 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:37.610985 (+ 1125us) log.cc:842] Serialized 26 byte log entry
0927 23:31:38.030208 (+419223us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:38.088138 (+ 57930us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.151975 (+ 63837us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.152880 (+ 905us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:38.153154 (+ 274us) raft_consensus.cc:1671] finished
0927 23:31:38.153169 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.153517 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50852,"prepare.queue_time_us":151710,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":85487,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":214,"thread_start_us":519904,"threads_started":3,"wal-append.queue_time_us":369080}
I20260927 23:31:38.310041 1656 rpcz_store.cc:276] 0927 23:31:37.289127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.289613 (+ 486us) service_pool.cc:227] Handling call
0927 23:31:38.309151 (+1019538us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b540013b0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:38.309306 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":561930624}
I20260927 23:31:38.277473 4524 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.354254 1660 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 23:31:38.356019 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.366025 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.269044 4301 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:38.368563 4495 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:38.352856 4287 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.398142 1654 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 9: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:38.408303 1734 raft_consensus.cc:2354] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:38.395555 4376 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:38.394125 4497 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:38.410945 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 84062us
I20260927 23:31:38.421526 4485 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.422134 4485 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.423632 4531 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.423288 4528 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.424427 4485 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.425514 4433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46178 (request call id 907) took 3325 ms. Trace:
I20260927 23:31:38.425722 4528 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.426074 4433 rpcz_store.cc:276] 0927 23:31:35.156632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:35.168744 (+ 12112us) service_pool.cc:227] Handling call
0927 23:31:35.715951 (+547207us) ts_tablet_manager.cc:1145] Deleting tablet 0be2a211b7754af89c38527623e5d967
0927 23:31:35.761786 (+ 45835us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0927 23:31:37.477248 (+1715462us) tablet_metadata.cc:641] Metadata flushed
0927 23:31:38.138392 (+661144us) tablet_metadata.cc:641] Metadata flushed
0927 23:31:38.425387 (+286995us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":298983,"tablet-delete.queue_time_us":391291,"thread_start_us":290356,"threads_started":1}
I20260927 23:31:38.427589 4389 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.404423 4441 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.428507 4524 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.430238 4441 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.430776 1657 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:38.407680 4444 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.435007 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:38.435642 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.438879 4376 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.442622 2257 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:38.443037 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.443679 4422 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.444103 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.444319 4422 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.445706 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.384330 4209 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.421962 4494 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.466776 1733 raft_consensus.cc:2354] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:38.467171 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:38.407464 4477 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.399499 4326 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.404232 4449 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.481300 4449 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.481379 4522 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.481295 4505 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.482378 4522 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.482777 4505 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.483418 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.484617 4522 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.484850 4505 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.485571 4450 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.486537 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.487890 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.489058 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.468691 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 489) took 1259 ms. Trace:
I20260927 23:31:38.500151 2286 rpcz_store.cc:276] 0927 23:31:37.209587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.215114 (+ 5527us) service_pool.cc:227] Handling call
0927 23:31:37.258184 (+ 43070us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:37.445205 (+187021us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.468223 (+1023018us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b540071f2d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:38.468522 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":102210304}
I20260927 23:31:38.502877 4264 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.503520 4264 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.503796 4303 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.475204 4526 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.507930 4326 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.507829 4526 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.508356 1702 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:38.475685 2290 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:38.475843 4534 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.475898 4523 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.476425 4485 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:38.477536 4477 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.479065 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:38.498688 1661 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:38.499473 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.510026 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:38.522755 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 75149.5us
I20260927 23:31:38.518970 1733 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 8
I20260927 23:31:38.516759 2290 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 494) took 1125 ms. Trace:
I20260927 23:31:38.534904 4477 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.536935 4029 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.537535 4029 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.538264 1702 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:38.538586 1580 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:38.539793 4029 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.540319 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.541252 4389 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:38.541810 4389 raft_consensus.cc:686] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:38.542549 4438 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:38.543216 4438 raft_consensus.cc:686] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:38.544147 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.545686 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.521586 1662 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.525493 2290 rpcz_store.cc:276] 0927 23:31:37.391353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.396549 (+ 5196us) service_pool.cc:227] Handling call
0927 23:31:38.516527 (+1119978us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b540059f454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:38.516677 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1629324672}
I20260927 23:31:38.555140 4486 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.555455 1730 raft_consensus.cc:1261] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 23:31:38.510700 4205 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.513042 4524 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.570724 4389 consensus_queue.cc:237] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.519788 4249 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:38.570883 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.555990 1629 leader_election.cc:398] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:38.579996 1735 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:38.597323 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.599869 4517 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:38.602721 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.603026 4475 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.583794 1629 leader_election.cc:401] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:38.576989 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1480) took 1861 ms. Trace:
W20260927 23:31:38.611989 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 56157.1us
I20260927 23:31:38.613809 1629 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:38.612330 1588 rpcz_store.cc:276] 0927 23:31:36.715606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.494141 (+778535us) service_pool.cc:227] Handling call
0927 23:31:37.494868 (+ 727us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:37.584362 (+ 89494us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:37.584405 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:37.584926 (+ 521us) log.cc:842] Serialized 26 byte log entry
0927 23:31:38.032587 (+447661us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:38.106348 (+ 73761us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.161143 (+ 54795us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.518511 (+357368us) spinlock_profiling.cc:243] Waited 957 us on lock 0x7b18003f0a60. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:38.556247 (+ 37736us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b5400466754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:38.556297 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:38.576633 (+ 20336us) raft_consensus.cc:1671] finished
0927 23:31:38.576654 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.576815 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66074,"prepare.queue_time_us":163462,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"raft.queue_time_us":70027,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":212,"spinlock_wait_cycles":124347648,"thread_start_us":518291,"threads_started":2,"wal-append.queue_time_us":389028}
I20260927 23:31:38.614511 4440 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.615459 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.510286 4526 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.511773 4483 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.514581 4528 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:38.616822 1588 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.618031 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.618532 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:38.619221 1735 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:38.619810 4136 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:38.624183 4209 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.586652 4249 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:38.627252 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.630456 1661 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:38.631759 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.632647 1587 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:38.633020 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.633304 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.515085 4523 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.581120 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.634680 1582 raft_consensus.cc:2379] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 9: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:38.635401 4523 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.588011 4205 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.637274 4205 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.549973 4485 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:38.637708 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.638451 4513 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.639043 2259 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:38.639868 4422 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.638492 4485 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.641260 4050 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:38.641651 4050 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:38.642091 4050 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.644105 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.601948 4263 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.590130 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.590437 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1481) took 1874 ms. Trace:
I20260927 23:31:38.645933 2293 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:38.646148 1583 rpcz_store.cc:276] 0927 23:31:36.715963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.495077 (+779114us) service_pool.cc:227] Handling call
0927 23:31:37.560426 (+ 65349us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:37.612204 (+ 51778us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.590074 (+977870us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b5400242050. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:38.590266 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144743808}
I20260927 23:31:38.590432 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.591428 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.647926 2291 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:38.647984 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.591581 2286 raft_consensus.cc:1261] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:38.578802 4494 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.650804 4494 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.515010 4534 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.551523 4326 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:38.652271 4326 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.593743 1663 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:38.576099 4372 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.018s
I20260927 23:31:38.653877 4444 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:31:38.654773 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.655076 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.655702 1581 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:38.656175 4326 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.616633 4528 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:38.656852 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1482) took 1932 ms. Trace:
I20260927 23:31:38.582448 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:38.657917 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:38.658064 4528 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.510805 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 0be2a211b7754af89c38527623e5d967 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:31:38.633471 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.587239 1662 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:38.659873 1470 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 149223us
I20260927 23:31:38.660121 4495 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.660969 1587 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 9: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:38.661260 1734 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:38.661510 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:38.662736 4495 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:38.652648 4534 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:38.572023 4438 consensus_queue.cc:237] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.595072 4310 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.663046 4489 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:38.679098 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.510632 4264 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:38.657847 4546 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260927 23:31:38.528016 4303 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.657522 1733 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:38.633431 4221 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:38.752763 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.657188 1585 rpcz_store.cc:276] 0927 23:31:36.724106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:37.498221 (+774115us) service_pool.cc:227] Handling call
0927 23:31:37.498584 (+ 363us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:37.582625 (+ 84041us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:37.582668 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:37.583355 (+ 687us) log.cc:842] Serialized 52 byte log entry
0927 23:31:38.022235 (+438880us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:38.023156 (+ 921us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.023746 (+ 590us) log.cc:842] Serialized 14 byte log entry
0927 23:31:38.311353 (+287607us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b180041de20. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db47638 00007f974db4ab11 00007f974db4aac9 00007f974db4aa59 00007f974db4aa21 00007f974db49d1d 00007f9756dab204 00007f9756dab039 00007f974db29dc6
0927 23:31:38.603937 (+292584us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b180041de20. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db14f96 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc
0927 23:31:38.623528 (+ 19591us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b480024c258. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97559167f1 00007f975590d8e7 00007f975590d77a 00007f9755a0195f 00007f9755a01627 00007f9755a15b2c 00007f9755a15ad8 00007f9755a15a58 00007f9755a15a13 00007f9755a14cbc 00007f9754996be6
0927 23:31:38.656315 (+ 32787us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b180041de20. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f97506fe82c 00007f974db299dc 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db44155 00007f9750a673d0 00007f9750a79576 00007f9750a8a0d7
0927 23:31:38.656417 (+ 102us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.656529 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:38.656583 (+ 54us) raft_consensus.cc:1671] finished
0927 23:31:38.656592 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:38.656713 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40440,"prepare.queue_time_us":334869,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":166370,"raft.run_cpu_time_us":2665,"raft.run_wall_time_us":598144,"spinlock_wait_cycles":155834112,"thread_start_us":488168,"threads_started":2,"wal-append.queue_time_us":378704}
I20260927 23:31:38.968756 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:38.961809 1554 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:38.759230 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.809092 1588 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:38.776937 1732 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:38.961519 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:38.777854 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 114686us
I20260927 23:31:38.719080 4489 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:38.835546 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 115263us
I20260927 23:31:38.984834 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:38.990207 4264 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:38.991030 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.990933 4264 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:38.992513 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.993181 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.994681 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:38.995280 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 96433.3us
W20260927 23:31:38.996034 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 59450.2us
I20260927 23:31:38.996819 4534 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:38.997270 4534 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:38.997651 4534 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.998337 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:38.998538 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:38.998659 4205 consensus_queue.cc:1035] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 23:31:39.000589 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 122979us
I20260927 23:31:39.000885 4494 consensus_queue.cc:1035] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:39.002701 4326 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 23:31:39.002954 1654 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 23:31:39.003489 4441 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:39.004382 1654 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:39.004177 4441 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.006863 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.008044 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:39.008816 1657 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:39.012184 1700 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:39.012805 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.014456 1662 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:39.015929 4050 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.011152 4441 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.018462 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.020984 1736 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:39.018003 4050 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.025455 4389 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:39.030448 1551 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.032106 4263 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:39.033253 4422 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 23:31:39.034438 4513 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:39.034864 4513 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:39.035564 4495 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:39.035753 4513 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.037671 4513 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:39.038357 4440 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:39.037341 4495 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.034862 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.040637 4326 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.043071 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.029249 4389 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.045506 1588 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:38.994625 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:38.980965 4301 consensus_meta.cc:287] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.575s user 0.009s sys 0.000s
I20260927 23:31:39.046651 1580 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
W20260927 23:31:39.050916 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 55043.9us
I20260927 23:31:39.052402 1659 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:38.994108 4264 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:38.984719 4489 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:38.961861 1486 catalog_manager.cc:5495] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 5 to 7, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 7 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:39.056427 4496 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.056677 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.057518 1656 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:39.064833 4326 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.101162 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.101262 4326 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:39.069314 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 70774.5us
I20260927 23:31:39.070647 2258 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:39.074486 4441 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:39.091254 4502 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:39.105696 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.091612 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.093814 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.094563 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.094504 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.107069 1581 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:39.048197 4221 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:39.107143 1583 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:39.109826 1662 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260927 23:31:39.111202 1662 pending_rounds.cc:76] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 6
I20260927 23:31:39.111488 1662 pending_rounds.cc:97] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260927 23:31:39.111920 1662 raft_consensus.cc:2866] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:39.110225 4326 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.113508 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.114830 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.115451 1583 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:39.116451 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.116648 1587 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:39.117825 4502 raft_consensus.cc:3032] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:39.118350 4502 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:39.118837 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.119536 1581 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:39.117733 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.122521 4494 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:39.122987 4494 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:39.123380 4494 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:39.128472 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:39.051740 4397 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.120364 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.133838 1581 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:38.780535 4310 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.135874 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.139048 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.139140 4422 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:39.139606 4500 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.152881 1657 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:39.157232 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.055984 4389 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.147351 4397 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.163825 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.164086 1583 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.164994 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.150676 1486 catalog_manager.cc:5495] T 87786104826046e7852edc590e194f5d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.151947 4495 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:39.166123 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:39.169081 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 98575.3us
I20260927 23:31:39.170706 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.185235 1551 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.179808 1587 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:39.186836 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:39.156243 4534 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.192771 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.196344 1586 raft_consensus.cc:3037] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.196798 1736 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:39.136595 4326 raft_consensus.cc:2741] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260927 23:31:39.190403 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.204365 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:39.205520 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 65480.3us
I20260927 23:31:39.208523 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.215641 1580 raft_consensus.cc:2354] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:39.218744 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.174845 4441 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.120128 4502 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.225303 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:39.199880 4249 consensus_meta.cc:287] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.574s user 0.011s sys 0.001s
I20260927 23:31:39.231956 1582 raft_consensus.cc:2354] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:39.235399 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.239042 1733 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:39.162729 4422 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.219933 4385 consensus_queue.cc:1035] T b9fe23b5411b441fbd8792ef9dc4e99b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260927 23:31:39.241359 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.221675 4376 raft_consensus.cc:2781] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:39.243836 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.242727 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.242489 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.241173 4382 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260927 23:31:39.248927 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.251603 4422 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.252022 1627 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.252187 4422 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:39.252533 4422 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.252915 4422 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.253676 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.254423 4303 consensus_queue.cc:1046] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:39.254925 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.255538 4303 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:39.256063 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.221355 4326 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.258199 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.258570 4326 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.259647 1733 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:39.260473 4326 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.256182 4303 raft_consensus.cc:3032] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:39.261348 4303 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:39.261513 4502 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.262207 4303 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.264233 4303 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.266181 1486 catalog_manager.cc:5495] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.267726 2259 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.268302 1486 catalog_manager.cc:5495] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 8, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 8 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:39.269629 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.270273 1585 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:39.259647 4440 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:39.213696 4495 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:39.274745 1552 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.275425 1732 raft_consensus.cc:2454] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:39.269708 1486 catalog_manager.cc:5495] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 3 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.294097 1586 raft_consensus.cc:2454] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:39.242146 4372 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:39.286908 4209 raft_consensus.cc:2741] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260927 23:31:39.277308 4326 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:39.306216 4050 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:39.234864 4249 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.364467 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.365589 4366 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.123s
I20260927 23:31:39.375874 4263 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.376456 4263 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.320037 4400 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:39.385988 1586 raft_consensus.cc:1261] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260927 23:31:39.397147 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 74065us
I20260927 23:31:39.378309 4263 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.384004 4441 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.336623 4495 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.414273 1585 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:39.353451 4264 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:39.398810 4050 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:39.434778 1702 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.407362 4400 raft_consensus.cc:686] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:39.304734 4221 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.362959 4372 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.437040 4397 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.427587 4263 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.409855 4326 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.447594 4513 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.458225 4264 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.441700 4489 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.446384 4534 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.450970 4310 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.472782 1583 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:39.475942 4489 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:31:39.467344 4263 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.484165 4263 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.495085 1702 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.473973 4500 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.508147 1702 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:39.512074 4549 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20260927 23:31:39.513082 4551 log.cc:924] Time spent T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260927 23:31:39.524896 4555 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260927 23:31:39.527086 4326 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.516731 4301 raft_consensus.cc:3032] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:39.528748 1486 catalog_manager.cc:5495] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 5 to 6, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 6 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.519325 4209 raft_consensus.cc:2726] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.519658 4249 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.528010 4494 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.498425 4050 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.531538 4301 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:39.549314 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.021s sys 0.019s
W20260927 23:31:39.549721 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.021s sys 0.019s
I20260927 23:31:39.530431 4500 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 23:31:39.550063 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 552046us
W20260927 23:31:39.550437 4485 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.022s sys 0.017s
I20260927 23:31:39.563553 4326 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.549437 4528 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:39.566058 1655 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.013s sys 0.031s
W20260927 23:31:39.583154 1655 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.583s user 0.013s sys 0.031s
I20260927 23:31:39.583647 4326 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:39.579032 4485 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.022s sys 0.017s
W20260927 23:31:39.586668 4554 log.cc:924] Time spent T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.118s user 0.003s sys 0.002s
I20260927 23:31:39.586426 4528 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.572646 4376 raft_consensus.cc:686] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:39.591081 1486 catalog_manager.cc:5495] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 3 to 6, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 6 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.594916 4495 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:39.556130 4422 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:39.581349 4548 log.cc:924] Time spent T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.182s user 0.000s sys 0.005s
I20260927 23:31:39.596760 4301 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.575071 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1583) took 1038 ms. Trace:
I20260927 23:31:39.607750 1584 rpcz_store.cc:276] 0927 23:31:38.537142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:38.537296 (+ 154us) service_pool.cc:227] Handling call
0927 23:31:38.537686 (+ 390us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:38.538009 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 23:31:38.538372 (+ 363us) log.cc:842] Serialized 14 byte log entry
0927 23:31:39.191011 (+652639us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800349b40. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974dabe4ca 00007f975719ff99 00007f974d982990 00007f974db44864 00007f974db44155 00007f9754444167 00007f9750a79807 00007f9750a88f0f 00007f9750a88e88
0927 23:31:39.574545 (+383534us) log.cc:842] Serialized 14 byte log entry
0927 23:31:39.574719 (+ 174us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:39.574765 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:39.574807 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:39.574869 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:39.575005 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":866,"prepare.queue_time_us":366891,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":80,"raft.queue_time_us":82017,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":20175,"spinlock_wait_cycles":26307968,"thread_start_us":597035,"threads_started":2,"wal-append.queue_time_us":892125}
W20260927 23:31:39.575480 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 390297us
I20260927 23:31:39.516052 4400 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.565732 1628 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:39.566785 4438 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.009s sys 0.033s
I20260927 23:31:39.628336 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.632337 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.633709 4494 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.635625 1581 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:39.636922 4422 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.637631 4301 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:39.623382 4372 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.638154 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.626921 1579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 451) took 1053 ms. Trace:
I20260927 23:31:39.640105 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.640973 1587 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:39.644312 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.645756 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.668380 1658 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:39.626456 4264 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.672364 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.624161 4376 consensus_queue.cc:237] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:39.626255 4326 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:39.650194 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.652381 4528 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:39.631172 1486 catalog_manager.cc:5495] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.655336 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:39.655591 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 89957.5us
I20260927 23:31:39.657234 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:39.702466 1654 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:39.703338 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:39.657527 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 382854us
W20260927 23:31:39.658942 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 416979us
W20260927 23:31:39.656056 4438 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.010s sys 0.033s
I20260927 23:31:39.670485 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:39.673539 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.696171 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:39.697561 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 445613us
I20260927 23:31:39.662936 4136 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:39.700829 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:39.696097 1663 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:39.666719 1579 rpcz_store.cc:276] 0927 23:31:38.573629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:38.577433 (+ 3804us) service_pool.cc:227] Handling call
0927 23:31:38.577881 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:38.578277 (+ 396us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:38.578330 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:38.578858 (+ 528us) log.cc:842] Serialized 26 byte log entry
0927 23:31:38.873311 (+294453us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:38.873388 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:38.873569 (+ 181us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:39.626614 (+753045us) raft_consensus.cc:1671] finished
0927 23:31:39.626631 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:39.626749 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6001,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":50549,"thread_start_us":293913,"threads_started":1,"wal-append.queue_time_us":810983}
I20260927 23:31:39.706511 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:39.711136 1662 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:39.711474 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.711716 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.712386 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.715528 2257 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.726070 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.728032 1729 raft_consensus.cc:2354] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:39.728246 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:39.729295 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.731498 1551 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.725142 4372 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260927 23:31:39.714859 4264 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.737347 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.738574 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.700350 4422 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:39.736042 1486 catalog_manager.cc:5495] T 4cfb0bd166774a248820c1b90412697d P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 3 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.741238 1654 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:39.743695 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.744024 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.746336 1486 catalog_manager.cc:5495] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 4 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:39.743131 1551 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.749155 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.750070 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.750782 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.752206 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.754487 1658 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:39.756048 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:39.756779 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:39.757491 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 489826us
I20260927 23:31:39.758890 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:39.738515 4485 raft_consensus.cc:2781] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:39.795568 2286 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:39.760318 1627 leader_election.cc:398] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:39.761286 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.766647 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:39.770610 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 65289.2us
I20260927 23:31:39.744288 4422 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:39.777658 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.798767 1552 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.799968 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:39.786676 1656 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:39.790481 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:39.792589 1486 catalog_manager.cc:5495] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.744978 4494 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:39.749395 4326 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:39.802531 1587 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:39.803390 1657 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:39.804639 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.806205 1551 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:39.804559 1554 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.786656 4565 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:39.810396 1656 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:39.808794 2258 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.019s sys 0.056s
W20260927 23:31:39.842056 2258 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s user 0.019s sys 0.056s
W20260927 23:31:39.840417 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 80326.5us
W20260927 23:31:39.842487 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 612300us
I20260927 23:31:39.803483 1486 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:39.843170 2258 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.842671 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.802516 4422 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:39.847661 1657 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:39.851476 1486 catalog_manager.cc:5495] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 5 to 7, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1). New cstate: current_term: 7 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:39.856860 1552 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.805444 4326 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.864951 1486 catalog_manager.cc:5495] T 99c7c13b00be47d38145b16ad8a788cc P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.843994 4494 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:39.853626 4513 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:39.898933 4564 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:39.899504 4564 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:39.846184 4502 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.899977 4568 consensus_queue.cc:1046] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260927 23:31:39.900580 4502 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:39.901518 4568 raft_consensus.cc:3032] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:39.901985 4568 raft_consensus.cc:729] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:39.894560 4397 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:39.902747 4567 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:39.903157 1486 catalog_manager.cc:5495] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 4 to 5, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 5 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.802028 4495 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20260927 23:31:39.878167 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 370089us
I20260927 23:31:39.811213 4528 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260927 23:31:39.916267 1702 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.875033 4326 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:39.900005 4564 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:39.925287 1702 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:39.944814 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 141189us
I20260927 23:31:39.918411 4397 raft_consensus.cc:686] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:39.950636 1702 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.950613 1486 catalog_manager.cc:5495] T 6e4c2a41313d4adcb6e037570a9d5882 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:39.924206 4534 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.951084 2259 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:39.927130 4209 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.926762 4496 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
W20260927 23:31:39.955744 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 104367us
W20260927 23:31:39.964082 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 120966us
I20260927 23:31:39.965425 1486 catalog_manager.cc:5495] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:31:39.971434 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 114668us
I20260927 23:31:39.961367 4050 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:39.960418 4496 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:39.952724 4572 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:39.973093 2258 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.976861 1552 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.980670 1552 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.984040 1552 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.984320 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.956017 4495 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260927 23:31:39.970062 4209 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:39.991005 1552 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.992497 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:39.955914 4567 consensus_queue.cc:1035] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
W20260927 23:31:39.999486 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 72892.3us
W20260927 23:31:39.986112 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 60227.8us
I20260927 23:31:40.001184 1702 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:39.966042 4500 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260927 23:31:39.988713 4376 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:39.987425 4485 raft_consensus.cc:686] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:39.991560 4209 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.000844 4389 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:39.989936 4050 raft_consensus.cc:686] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:40.094898 4534 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 23:31:40.070714 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 69626.9us
I20260927 23:31:39.988481 4263 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.098272 4570 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:39.978233 4502 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:40.099188 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 95434.6us
I20260927 23:31:40.001755 4571 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:40.005558 4397 consensus_queue.cc:237] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:40.104269 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 103745us
I20260927 23:31:40.083240 4582 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:39.989686 1486 catalog_manager.cc:5495] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 3, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:31:40.114982 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 122583us
I20260927 23:31:40.098898 4565 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:39.987272 4494 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:40.143586 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.125376 1702 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:40.116503 1482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48984 (request call id 243) took 20041 ms (client timeout 15000 ms). Trace:
I20260927 23:31:40.108121 4209 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:40.132611 4570 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.158138 4050 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.160317 4570 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.161985 4502 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:40.162842 4565 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.164150 4221 consensus_queue.cc:1046] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:40.164497 1627 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.165940 1627 leader_election.cc:398] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:40.166282 1627 leader_election.cc:401] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 23:31:40.110800 4389 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.172549 4310 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:40.173285 4389 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.139220 4422 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.175704 4422 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:40.147840 4574 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260927 23:31:40.175848 4568 consensus_queue.cc:260] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.176908 4568 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.171939 4494 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:40.150406 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 374) took 1153 ms. Trace:
I20260927 23:31:40.154579 4588 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:40.145987 1486 catalog_manager.cc:5495] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 reported cstate change: term changed from 2 to 4, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 4 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 23:31:40.147624 4575 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260927 23:31:40.166522 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.128923 4534 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.171015 4582 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:40.171175 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.171746 4301 raft_consensus.cc:3032] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:40.172004 4249 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.172748 4592 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:40.148373 1660 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:40.179735 1655 rpcz_store.cc:276] 0927 23:31:38.997219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:38.999286 (+ 2067us) service_pool.cc:227] Handling call
0927 23:31:38.999639 (+ 353us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:39.000065 (+ 426us) raft_consensus.cc:1520] Early marking committed up to index 8
0927 23:31:39.586032 (+585967us) log.cc:842] Serialized 14 byte log entry
0927 23:31:39.957214 (+371182us) log.cc:842] Serialized 14 byte log entry
0927 23:31:39.957947 (+ 733us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:39.958602 (+ 655us) log.cc:842] Serialized 26 byte log entry
0927 23:31:39.958751 (+ 149us) raft_consensus.cc:1631] Marking committed up to 8
0927 23:31:39.958793 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:39.958878 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:40.150152 (+191274us) raft_consensus.cc:1671] finished
0927 23:31:40.150174 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:40.150305 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":563903,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":76,"raft.queue_time_us":3990,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":5561,"spinlock_wait_cycles":14636928,"thread_start_us":911453,"threads_started":2,"wal-append.queue_time_us":376392}
I20260927 23:31:40.154762 4485 consensus_queue.cc:237] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.188581 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:40.193367 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 68094us
I20260927 23:31:40.194031 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.195029 4495 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:40.195137 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.195562 4495 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.186973 4589 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:40.147660 4578 log.cc:924] Time spent T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20260927 23:31:40.187520 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.196118 4589 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.187368 4582 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.196847 2286 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:40.188633 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.172560 4586 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.197925 2288 raft_consensus.cc:2354] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:40.197829 4586 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.198235 4582 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.190230 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.191136 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.191828 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.199849 4586 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.182793 4494 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.200531 4495 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:40.201663 4494 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.181521 4588 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.162575 4502 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:40.211315 1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46188 (request call id 1847) took 1579 ms (client timeout 1891 ms). Trace:
I20260927 23:31:40.211954 4502 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:40.211850 1661 rpcz_store.cc:269] 0927 23:31:38.632012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:38.654520 (+ 22508us) service_pool.cc:227] Handling call
0927 23:31:40.211078 (+1556558us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003f3754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:40.211229 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2963480448}
I20260927 23:31:40.208637 4249 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:40.184396 4301 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:40.213568 4326 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.185143 4438 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 23:31:40.213923 4301 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.185284 4422 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.215651 4301 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.182667 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.208575 4588 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.208967 4593 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:40.198235 4589 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.218343 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.219540 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:40.146018 1482 rpcz_store.cc:269] 0927 23:31:19.911295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:19.911581 (+ 286us) service_pool.cc:227] Handling call
0927 23:31:39.944420 (+20032839us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:37.969354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:37.982994 (+ 13640us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:37.983048 (+ 54us) write_op.cc:435] Acquired schema lock
0927 23:31:37.983068 (+ 20us) tablet.cc:660] Decoding operations
0927 23:31:37.984691 (+ 1623us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:37.984772 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:37.984818 (+ 46us) tablet.cc:683] Acquiring locks for 15 operations
0927 23:31:37.986614 (+ 1796us) tablet.cc:699] Row locks acquired
0927 23:31:37.986642 (+ 28us) write_op.cc:260] PREPARE: finished
0927 23:31:37.986843 (+ 201us) write_op.cc:270] Start()
0927 23:31:37.986991 (+ 148us) write_op.cc:276] Timestamp: P: 1790551897986775 usec, L: 0
0927 23:31:37.987014 (+ 23us) op_driver.cc:331] REPLICATION: starting
0927 23:31:37.989842 (+ 2828us) log.cc:842] Serialized 5809 byte log entry
0927 23:31:38.170339 (+180497us) op_driver.cc:444] REPLICATION: finished
0927 23:31:38.385372 (+215033us) write_op.cc:301] APPLY: starting
0927 23:31:38.394506 (+ 9134us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:38.485321 (+ 90815us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:38.485485 (+ 164us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:38.487363 (+ 1878us) spinlock_profiling.cc:243] Waited 61.1 ms on lock 0x7b5400341054. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6c689 00007f9750a0a804 00007f9750a0a7b3 00007f9750a0a728 00007f9750a0a6e3 00007f9750a099cc 00007f9750a33a16 00007f9750a1b5cb 00007f9750a05c8d 00007f9750a09665
0927 23:31:38.487530 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:38.499678 (+ 12148us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0927 23:31:38.499722 (+ 44us) write_op.cc:312] APPLY: finished
0927 23:31:38.500302 (+ 580us) log.cc:842] Serialized 107 byte log entry
0927 23:31:38.500518 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:38.500894 (+ 376us) write_op.cc:454] Released schema lock
0927 23:31:38.500970 (+ 76us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:38.501461 (+ 491us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:38.501486 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":859151,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":169153,"apply.run_cpu_time_us":6154,"apply.run_wall_time_us":132308,"num_ops":15,"prepare.queue_time_us":490629,"prepare.run_cpu_time_us":7290,"prepare.run_wall_time_us":186584,"raft.queue_time_us":1203,"raft.run_cpu_time_us":18304,"raft.run_wall_time_us":391648,"replication_time_us":183062,"spinlock_wait_cycles":140556160,"thread_start_us":598824,"threads_started":4,"wal-append.queue_time_us":1753,"wal-append.run_cpu_time_us":5947,"wal-append.run_wall_time_us":1528899}]]}
I20260927 23:31:40.220187 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.224529 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.234812 1736 raft_consensus.cc:2379] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:40.233315 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.235123 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.236030 1587 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:40.236847 1585 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:40.104661 4496 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.237780 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.237864 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.241097 1733 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:40.241406 1728 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.238656 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.238014 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.242125 1737 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.242662 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.243216 1729 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.244656 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:40.247454 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.249607 1702 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.252045 1579 raft_consensus.cc:1261] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:40.255584 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.257472 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.257954 1735 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:40.230978 4513 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:40.260774 4513 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.263329 4513 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.228149 4593 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.231536 4592 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.267503 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:40.271874 2294 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:40.269579 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:40.272024 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.272730 1655 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.239225 4502 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:40.274567 4502 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:40.274814 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.303728 1582 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:40.275522 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:40.304826 1586 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:40.278661 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.305770 1583 raft_consensus.cc:2354] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:40.280927 4593 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.286839 4513 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.287178 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.242851 4422 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.287285 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.319141 1735 raft_consensus.cc:2379] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:40.287989 4557 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.248257 4589 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:40.288462 4500 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:40.310786 4496 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.253911 4221 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:40.248668 4303 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.290750 4326 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.313599 4249 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.380286 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:40.233388 4376 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260927 23:31:40.233563 4582 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:40.292891 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.286118 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.297957 1702 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.302685 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.307804 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.317162 4564 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.321231 1737 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:40.351173 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.400287 1663 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:40.310796 4594 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:40.288774 1581 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:40.376056 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 127405us
I20260927 23:31:40.398933 1585 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
W20260927 23:31:40.402164 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 104291us
I20260927 23:31:40.386932 1729 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
W20260927 23:31:40.386991 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 353950us
I20260927 23:31:40.391222 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 0be2a211b7754af89c38527623e5d967 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:46178
W20260927 23:31:40.394418 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 85057.5us
I20260927 23:31:40.382373 4221 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:40.368753 4589 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.405786 1662 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:40.406508 1702 leader_election.cc:398] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:40.410002 1702 leader_election.cc:401] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:40.320835 4513 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:40.412307 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 44528.4us
I20260927 23:31:40.418689 1628 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.421270 1586 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 10.
I20260927 23:31:40.422915 1588 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 9. Current term is 10. Ops: []
I20260927 23:31:40.366390 4422 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.427825 2292 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:40.407990 4221 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:40.404466 4594 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.404870 4500 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:40.485034 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.488088 1737 raft_consensus.cc:3037] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.472769 4528 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.411892 4589 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:40.479000 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75899.9us
I20260927 23:31:40.482748 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:40.401623 4582 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.397049 4303 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.501248 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:40.510311 4599 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260927 23:31:40.503116 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.511776 1728 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.486109 4594 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:40.495713 4612 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.543206 4610 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.417948 4502 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.499341 4513 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.545882 1587 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:40.551461 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.553565 1728 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.517083 4563 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:40.546006 1470 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f662be9387d148199907fd0e049ba4be with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 1b945905616b4989853b0a92bfbd6b98 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:40.558151 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.564908 1655 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:40.545143 4500 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:40.579224 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:40.580819 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 178347us
I20260927 23:31:40.582006 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.501008 4528 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.583684 1663 raft_consensus.cc:1261] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:40.497851 4615 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:40.588020 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:40.551717 4612 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.593647 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.593338 2294 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
W20260927 23:31:40.595431 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 125600us
I20260927 23:31:40.580366 4610 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:40.600361 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.601007 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.601063 1551 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:40.604740 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:40.605023 4303 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.601475 1663 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:40.558548 4581 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260927 23:31:40.600176 1655 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.580013 4563 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.620110 4422 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.620234 1581 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 23:31:40.621934 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.622608 1734 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:40.630422 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.635775 1737 raft_consensus.cc:2454] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:40.585295 4495 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.597697 4617 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.619199 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.612510 4513 raft_consensus.cc:2781] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:40.675144 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:40.612625 4564 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:40.675843 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 116467us
I20260927 23:31:40.676661 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.679572 1734 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:40.680480 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 61397.7us
I20260927 23:31:40.681641 1551 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:40.682906 1551 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:40.686728 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.691533 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.594915 4588 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.692958 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:40.695101 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 63148.3us
I20260927 23:31:40.696863 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.674129 4422 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:40.698901 1655 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:40.677369 4496 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.704173 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.686937 4585 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:40.702881 4592 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:40.603550 4615 consensus_queue.cc:1035] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20260927 23:31:40.617825 4326 consensus_queue.cc:1035] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260927 23:31:40.675663 4600 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:40.697418 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.703229 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.727550 1737 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:40.707597 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:40.717029 4209 consensus_meta.cc:287] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.001s
W20260927 23:31:40.706936 4603 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20260927 23:31:40.694099 4565 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260927 23:31:40.747942 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50638.4us
W20260927 23:31:40.732750 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 489566us
I20260927 23:31:40.694783 4513 raft_consensus.cc:686] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:40.745327 1734 raft_consensus.cc:2379] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:40.746171 1655 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 23:31:40.698458 4617 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:40.748833 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.613119 4610 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.754348 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.756320 1552 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.758665 1552 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:40.760376 1730 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:40.756784 4397 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.745204 4582 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:40.763594 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.748270 4310 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:40.752445 4301 consensus_meta.cc:287] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.534s user 0.008s sys 0.002s
I20260927 23:31:40.752617 4592 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:40.764478 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.751287 4565 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.749931 4585 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:40.720089 4422 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.734879 4496 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.792888 1552 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.793100 1703 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.748766 4209 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.762174 4600 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:40.771217 1734 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.785873 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:40.797909 1731 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:40.764356 4263 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:40.790338 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:40.773388 4614 log.cc:924] Time spent T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20260927 23:31:40.801874 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.770710 4597 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
W20260927 23:31:40.792624 4389 consensus_meta.cc:287] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.619s user 0.011s sys 0.000s
I20260927 23:31:40.811040 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.794356 4615 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.790426 4565 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.807351 1627 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.808303 1730 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:40.773362 4598 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.773803 4310 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.810909 1733 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:40.770370 4586 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260927 23:31:40.811883 1731 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:40.756898 4617 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.801339 4612 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.801602 4326 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:40.797981 4397 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:40.816258 4622 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.810489 4301 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.859789 1663 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:40.797291 4600 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.811331 4528 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:40.866251 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 147116us
I20260927 23:31:40.835178 4310 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:40.823688 4597 raft_consensus.cc:686] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:40.877082 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 84308.3us
I20260927 23:31:40.832182 4495 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:40.872035 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.810878 4630 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:40.863101 4326 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.862902 4136 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:40.873222 1628 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.880302 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.870916 4588 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:40.881924 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 74676.7us
I20260927 23:31:40.816473 4389 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.878652 4528 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.896106 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.875902 4598 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260927 23:31:40.868563 4397 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:40.869676 4582 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:40.872816 4563 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.889554 4326 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:40.870244 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 77214.8us
I20260927 23:31:40.920817 1737 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:40.920951 1736 raft_consensus.cc:1261] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260927 23:31:40.837713 4615 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.898797 1628 leader_election.cc:398] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501) with higher term. Message: Invalid argument: T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6.
I20260927 23:31:40.817775 4513 consensus_queue.cc:237] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.903357 4301 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:40.926654 1703 leader_election.cc:398] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6.
I20260927 23:31:40.927623 1703 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:40.928407 4620 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20260927 23:31:40.928772 1703 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.919838 4582 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.932153 1628 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:40.903978 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:40.900774 4640 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:40.906311 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:40.906661 4635 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:40.907972 4597 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:40.911018 4617 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:40.889346 4495 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.872766 4600 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:40.901219 4397 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.933061 1736 raft_consensus.cc:1261] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:40.894335 4263 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:40.963102 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.942068 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.942533 1587 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:40.962773 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.943964 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.965404 1585 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:40.965613 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:40.946482 4586 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:40.938282 1733 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:40.950845 1628 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
W20260927 23:31:40.951490 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 376316us
I20260927 23:31:40.885529 4585 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:40.955554 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 51678.6us
I20260927 23:31:40.977463 1737 raft_consensus.cc:1261] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:40.956523 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 375061us
I20260927 23:31:40.965735 1583 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:40.944669 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:40.966269 1587 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:40.955308 4635 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.953788 4389 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:40.973263 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:40.974700 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:40.975334 4631 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260927 23:31:40.975668 4586 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.981663 1581 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.982204 1470 catalog_manager.cc:4841] TS 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051): tablet 0be2a211b7754af89c38527623e5d967 (table test-workload [id=256c41e6bcc14d84a97ed516e1cf9e24]) successfully deleted
I20260927 23:31:40.982920 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:40.951020 4640 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:40.877352 4592 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:40.974745 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:40.950641 4438 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:40.988104 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:41.011448 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:40.962360 4264 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:41.012099 1584 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:40.917366 4528 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.014561 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:40.964552 4263 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:40.979938 4496 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:40.988968 1735 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 23:31:41.047447 4637 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 23:31:40.979797 4209 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:40.889038 4630 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.014850 4605 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260927 23:31:41.133476 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:41.005000 4640 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:41.142742 1587 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
W20260927 23:31:41.130652 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 157495us
W20260927 23:31:41.130925 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 104914us
W20260927 23:31:41.133287 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 150451us
I20260927 23:31:41.133698 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:41.020131 4264 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:41.071218 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:41.084254 1581 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:41.144397 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:41.146186 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:41.019055 4389 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:41.022595 4263 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:41.074836 4444 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.012s sys 0.026s
I20260927 23:31:41.152920 1587 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:41.096057 1585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:41.173452 1585 raft_consensus.cc:595] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:41.020574 4422 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:41.161473 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:41.127873 4619 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:41.181794 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.173803 1585 raft_consensus.cc:603] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:41.173807 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:41.174249 2258 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:41.175105 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.096858 4637 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:41.182746 1702 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:41.151538 4264 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:41.149052 4598 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:41.074848 4495 consensus_queue.cc:1046] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.103s
I20260927 23:31:41.121984 4610 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:41.191695 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.192144 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:41.148996 4605 consensus_queue.cc:1225] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Peer 0fe98f486bab4d0b9c421c145d83f1ce log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260927 23:31:41.172976 4444 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.884s user 0.016s sys 0.030s
I20260927 23:31:41.165794 4209 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:41.200248 4209 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:41.200680 4209 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:41.201483 4595 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 23:31:41.193542 4598 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:41.202749 4598 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:41.202785 4528 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:41.181819 4619 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:41.204852 4619 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:41.206082 1585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:41.191226 4604 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.007s
I20260927 23:31:41.207108 4643 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.207700 4643 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.200083 4637 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.183295 4582 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:41.186147 4389 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:41.272024 4389 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:41.242566 1734 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:41.212008 4502 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:41.247105 4568 consensus_meta.cc:287] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.070s user 0.013s sys 0.000s
W20260927 23:31:41.247542 4589 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.720s user 0.010s sys 0.001s
I20260927 23:31:41.196554 4564 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:41.188237 4557 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
W20260927 23:31:41.268839 4221 consensus_meta.cc:287] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.770s user 0.006s sys 0.006s
I20260927 23:31:41.242761 4366 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:41.287949 1728 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:41.197793 4572 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:41.268220 4635 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:41.191706 4422 consensus_queue.cc:1035] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
I20260927 23:31:41.281757 4389 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:41.292416 4496 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.262008 4569 consensus_queue.cc:1035] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.036s
I20260927 23:31:41.292677 4586 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:41.330255 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 347033us
I20260927 23:31:41.300213 4615 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:41.304268 4572 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:41.288409 4589 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.350492 2287 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 23:31:41.502194 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.504374 2288 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:41.506453 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.294719 4221 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:41.565785 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 418478us
I20260927 23:31:41.567919 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:41.299721 4649 consensus_queue.cc:1035] T 2b90254fab234e29ba132c165c446490 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.217s
I20260927 23:31:41.602859 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:41.600485 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:41.601464 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.604115 1729 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:41.500888 4643 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:41.609429 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:41.621908 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.357739 4579 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.248s
I20260927 23:31:41.632889 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:41.626791 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:41.626969 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:41.345594 4496 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.635102 1654 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:41.632550 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:41.636396 1660 raft_consensus.cc:2379] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 9: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:41.601702 4641 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:41.610617 4628 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.649727 1629 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:41.615339 4500 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.014s sys 0.020s
I20260927 23:31:41.610774 4592 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.617556 4563 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.651405 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.668329 1657 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 23:31:41.659598 4500 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.014s sys 0.020s
W20260927 23:31:41.672386 1584 consensus_meta.cc:287] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.655s user 0.010s sys 0.000s
I20260927 23:31:41.685546 1662 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:41.676848 1654 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:41.685160 1584 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:41.678637 4563 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.690691 4563 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:41.691365 4563 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 23:31:41.694693 4221 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:41.703045 4310 consensus_meta.cc:287] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.819s user 0.007s sys 0.003s
I20260927 23:31:41.703851 4310 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.730988 4612 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:41.731390 4612 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:41.732124 4612 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:41.733209 4600 consensus_meta.cc:287] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.756s user 0.004s sys 0.004s
I20260927 23:31:41.671815 4592 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.735118 4592 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:41.735634 4630 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:41.736912 4593 consensus_meta.cc:287] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.367s user 0.012s sys 0.000s
I20260927 23:31:41.737313 4593 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.739377 4593 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:41.714901 4645 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.363s user 0.000s sys 0.001s
W20260927 23:31:41.716392 4627 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.922s user 0.000s sys 0.004s
W20260927 23:31:41.718796 4618 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.942s user 0.006s sys 0.019s
I20260927 23:31:41.720100 4485 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 23:31:41.720737 4303 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.788980 4654 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:41.788895 4303 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.789580 4654 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:41.725926 4565 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.889s user 0.008s sys 0.000s
I20260927 23:31:41.726634 4659 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:41.728333 4209 consensus_meta.cc:287] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.527s user 0.006s sys 0.007s
I20260927 23:31:41.702903 4610 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:41.686522 4500 raft_consensus.cc:686] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:41.733255 4612 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:41.731344 4310 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:41.743755 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1734) took 1165 ms. Trace:
I20260927 23:31:41.661007 4569 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:41.750844 1703 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:41.752485 4605 raft_consensus.cc:3032] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Stepping down as leader of term 7
W20260927 23:31:41.757045 4657 log.cc:924] Time spent T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260927 23:31:41.758778 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:41.760116 4647 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.486s user 0.000s sys 0.001s
I20260927 23:31:41.661556 4628 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:41.764683 4652 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
W20260927 23:31:41.766636 4326 consensus_meta.cc:287] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.816s user 0.009s sys 0.001s
I20260927 23:31:41.669031 4641 raft_consensus.cc:686] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:41.710841 4653 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.349s user 0.000s sys 0.001s
I20260927 23:31:41.846176 4610 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.841629 4209 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.851583 4630 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 23:31:41.767200 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 162045us
W20260927 23:31:41.704311 4651 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260927 23:31:41.697935 4264 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:41.850680 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 99925.9us
I20260927 23:31:41.846074 1728 rpcz_store.cc:276] 0927 23:31:40.578803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.578959 (+ 156us) service_pool.cc:227] Handling call
0927 23:31:41.743555 (+1164596us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54002ee0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:41.743672 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628094848}
I20260927 23:31:41.651535 4572 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.790603 4485 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:41.778903 4600 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.855746 1734 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:42.151016 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1618) took 1207 ms. Trace:
I20260927 23:31:41.951336 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:41.939113 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:42.183970 1581 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:41.957299 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 308891us
W20260927 23:31:42.126865 2257 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.000s sys 0.017s
I20260927 23:31:42.186712 1703 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:41.939007 1552 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:41.896358 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:42.188352 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 249445us
W20260927 23:31:41.981385 1627 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.005s sys 0.004s
I20260927 23:31:42.006562 1661 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 LEADER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 9. Current term is 10. Ops: []
W20260927 23:31:42.189450 1627 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.039s user 0.012s sys 0.007s
W20260927 23:31:42.189846 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 1.0517e+06us
W20260927 23:31:42.127200 1588 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.006s sys 0.010s
W20260927 23:31:42.190893 1588 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.643s user 0.007s sys 0.011s
I20260927 23:31:41.841374 4593 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.938660 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:42.187695 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 224668us
I20260927 23:31:41.840385 4659 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.795192 4565 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.189476 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:42.328934 1582 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:42.309660 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 119157us
I20260927 23:31:42.285444 1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 444) took 1528 ms. Trace:
W20260927 23:31:42.330745 2257 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.003s sys 0.020s
W20260927 23:31:42.328496 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 141842us
I20260927 23:31:42.332302 1703 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:42.182583 1586 rpcz_store.cc:276] 0927 23:31:40.943551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.943627 (+ 76us) service_pool.cc:227] Handling call
0927 23:31:40.970340 (+ 26713us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:41.147782 (+177442us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:41.147823 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:41.148279 (+ 456us) log.cc:842] Serialized 26 byte log entry
0927 23:31:41.181083 (+ 32804us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:41.181126 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:41.183814 (+ 2688us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:41.756304 (+572490us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:42.150632 (+394328us) raft_consensus.cc:1671] finished
0927 23:31:42.150662 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:42.150945 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83607,"raft.queue_time_us":41549,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5700,"spinlock_wait_cycles":5667840,"thread_start_us":196935,"threads_started":2,"wal-append.queue_time_us":170585}
I20260927 23:31:42.333470 1661 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260927 23:31:42.329680 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 140276us
W20260927 23:31:42.126590 1628 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.010s sys 0.007s
W20260927 23:31:42.338080 1628 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.191s user 0.018s sys 0.009s
W20260927 23:31:42.338819 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 1.19222e+06us
I20260927 23:31:42.339689 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.344566 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:42.346105 1734 raft_consensus.cc:1261] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260927 23:31:42.152149 4502 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.865s user 0.009s sys 0.001s
W20260927 23:31:42.137094 4397 consensus_meta.cc:287] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 1.128s user 0.003s sys 0.006s
I20260927 23:31:42.003353 4209 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:42.146185 4389 consensus_meta.cc:287] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.791s user 0.007s sys 0.000s
I20260927 23:31:41.854238 4326 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.913138 4589 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.395062 4502 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:42.403539 4502 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.404189 4326 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.404785 1585 raft_consensus.cc:595] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:42.346662 4593 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.405129 1585 raft_consensus.cc:603] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:41.926962 4630 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:42.405563 4502 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.405797 4630 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.322079 4622 consensus_queue.cc:1046] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260927 23:31:42.406880 4622 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:42.407207 4622 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:41.953365 1554 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.4:45509 timed out after 1.650s (SENT) [suppressed 7 similar messages]
I20260927 23:31:42.407330 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.981513 4595 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:42.408165 4595 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:42.408911 2260 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.005s sys 0.022s
I20260927 23:31:42.408641 4397 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:42.409283 2260 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s user 0.005s sys 0.022s
I20260927 23:31:42.141033 4500 consensus_queue.cc:237] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:41.854875 4641 consensus_queue.cc:237] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.410513 4397 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.284436 4628 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.275480 4600 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:42.090138 4598 consensus_meta.cc:287] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.819s user 0.010s sys 0.002s
W20260927 23:31:42.171607 4263 consensus_meta.cc:287] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.990s user 0.003s sys 0.007s
I20260927 23:31:42.330787 1730 rpcz_store.cc:276] 0927 23:31:40.765527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.930394 (+164867us) service_pool.cc:227] Handling call
0927 23:31:40.930647 (+ 253us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:40.930948 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:40.930981 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:40.931379 (+ 398us) log.cc:842] Serialized 26 byte log entry
0927 23:31:41.149677 (+218298us) raft_consensus.cc:1631] Marking committed up to 8
0927 23:31:41.149717 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:41.149787 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:42.285200 (+1135413us) raft_consensus.cc:1671] finished
0927 23:31:42.285223 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:42.285355 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10392,"raft.queue_time_us":21050,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":3459,"thread_start_us":217944,"threads_started":1,"wal-append.queue_time_us":388202}
I20260927 23:31:42.352995 1581 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:42.191398 4310 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:42.399008 1732 raft_consensus.cc:2396] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260927 23:31:42.404104 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 75750.4us
W20260927 23:31:42.155447 4640 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.982s user 0.010s sys 0.000s
W20260927 23:31:42.149123 4594 consensus_meta.cc:287] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.598s user 0.011s sys 0.000s
I20260927 23:31:42.386096 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:42.394045 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.191437 4496 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.397769 4646 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:42.386600 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:42.398821 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.420068 4589 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 23:31:42.416754 1551 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:42.152961 4619 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.948s user 0.009s sys 0.002s
I20260927 23:31:42.464699 1731 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260927 23:31:42.465225 1731 pending_rounds.cc:76] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 4
I20260927 23:31:42.465472 1731 pending_rounds.cc:97] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260927 23:31:42.465747 1731 raft_consensus.cc:2866] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:42.418506 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:42.466408 1735 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:42.385172 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:42.140923 4303 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.423271 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:41.860446 4569 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:41.851352 4605 raft_consensus.cc:729] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:42.432890 4636 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.411446 4389 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.426775 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:42.487528 1581 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:42.432159 4600 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:41.857795 4610 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.341574 4656 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:42.138818 4637 raft_consensus.cc:482] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:42.191569 4669 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:42.341862 4301 consensus_queue.cc:1046] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260927 23:31:42.458304 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:42.014712 4618 log.cc:924] Time spent T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260927 23:31:42.485877 4597 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:42.461556 1582 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:42.427877 4630 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.419881 4397 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:42.433813 4640 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.462662 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:42.457911 4594 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.425508 4502 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:42.432605 4647 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260927 23:31:42.284204 4264 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.470930 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47570 (request call id 31) took 1254 ms. Trace:
I20260927 23:31:42.523052 1585 rpcz_store.cc:276] 0927 23:31:41.205783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:41.205941 (+ 158us) service_pool.cc:227] Handling call
0927 23:31:42.459861 (+1253920us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003093d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a69bc3 00007f97559b381a 00007f974f0a6d04 00007f974f0a6c92 00007f974f0a6bc1 00007f974f0a6b3c 00007f974f0a5df2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:42.459968 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2755344640}
W20260927 23:31:42.478284 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 61620.4us
I20260927 23:31:42.478417 4619 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:42.479602 1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1727) took 2208 ms (client timeout 2703 ms). Trace:
I20260927 23:31:42.529541 1661 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:42.525295 1732 rpcz_store.cc:269] 0927 23:31:40.255781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.256281 (+ 500us) service_pool.cc:227] Handling call
0927 23:31:42.464034 (+2207753us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b54002be654. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:42.464196 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4600937728}
I20260927 23:31:42.516588 4586 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.490748 4636 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.385679 4612 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260927 23:31:42.426486 1552 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:42.424191 4598 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.462972 4622 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.467738 4654 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.498245 1586 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:42.458288 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:42.500656 1583 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:42.495894 4593 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.498569 4637 raft_consensus.cc:504] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.503444 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 501) took 2206 ms. Trace:
W20260927 23:31:42.508285 4681 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260927 23:31:42.514799 2290 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:42.469283 4646 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.547081 1656 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260927 23:31:42.551241 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.458389 4485 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.425781 4263 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.552604 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:42.542003 4683 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260927 23:31:42.497581 4600 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.519129 4397 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.541280 4586 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.631863 1657 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:42.622373 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1742) took 1676 ms. Trace:
I20260927 23:31:42.556041 1581 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:42.546005 4301 raft_consensus.cc:3032] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:42.601994 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:42.634290 1580 raft_consensus.cc:2379] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:42.591413 4619 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.593137 4636 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.630039 1735 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:42.561061 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.596256 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.640443 1731 rpcz_store.cc:276] 0927 23:31:40.945985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.977961 (+ 31976us) service_pool.cc:227] Handling call
0927 23:31:42.464299 (+1486338us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.480160 (+ 15861us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:42.622130 (+141970us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002be650. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:42.622302 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3351535488}
I20260927 23:31:42.633656 1656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 393) took 1018 ms. Trace:
I20260927 23:31:42.646930 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.655155 1657 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:42.655558 1656 rpcz_store.cc:276] 0927 23:31:41.614989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:41.615103 (+ 114us) service_pool.cc:227] Handling call
0927 23:31:41.637668 (+ 22565us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:42.632911 (+995243us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:42.633437 (+ 526us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b5400352354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:42.633599 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2090749696}
I20260927 23:31:42.643221 4654 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:42.651757 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.573798 4565 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.523568 4605 consensus_queue.cc:260] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.560737 4598 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.637036 4389 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.632778 2293 rpcz_store.cc:276] 0927 23:31:40.297006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.310152 (+ 13146us) service_pool.cc:227] Handling call
0927 23:31:40.370342 (+ 60190us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:40.407994 (+ 37652us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:40.408025 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:40.411998 (+ 3973us) log.cc:842] Serialized 26 byte log entry
0927 23:31:40.547189 (+135191us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:40.565084 (+ 17895us) log.cc:842] Serialized 14 byte log entry
0927 23:31:40.937167 (+372083us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b1800376060. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974dabe4ca 00007f975719ff99 00007f974d982990 00007f974db44864 00007f974db44155 00007f9750a673d0 00007f9750a79576 00007f9750a8a0d7 00007f9750a8a058
0927 23:31:40.937379 (+ 212us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:40.937469 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:41.718105 (+780636us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:42.503248 (+785143us) raft_consensus.cc:1671] finished
0927 23:31:42.503268 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:42.503372 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97688,"prepare.queue_time_us":26899,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":407894,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":7559,"spinlock_wait_cycles":83999360,"thread_start_us":54209,"threads_started":1,"wal-append.queue_time_us":167642}
W20260927 23:31:42.692332 1588 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.008s sys 0.040s
W20260927 23:31:42.692708 1588 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s user 0.008s sys 0.040s
I20260927 23:31:42.643738 4301 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:42.524049 4502 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.643951 4659 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.644366 4595 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.559211 4310 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.131s
I20260927 23:31:42.663898 4565 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.663870 4654 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:42.698916 4605 raft_consensus.cc:3037] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:42.705876 1737 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:42.721195 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 484) took 1393 ms. Trace:
I20260927 23:31:42.724548 1588 rpcz_store.cc:276] 0927 23:31:41.327879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:41.346644 (+ 18765us) service_pool.cc:227] Handling call
0927 23:31:41.494630 (+147986us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:41.494888 (+ 258us) raft_consensus.cc:1520] Early marking committed up to index 9
0927 23:31:42.191274 (+696386us) log.cc:842] Serialized 14 byte log entry
0927 23:31:42.720743 (+529469us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:42.720814 (+ 71us) raft_consensus.cc:1631] Marking committed up to 9
0927 23:31:42.720884 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:42.720964 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:42.721124 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27316,"prepare.queue_time_us":210519,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"thread_start_us":1144961,"threads_started":2}
I20260927 23:31:42.700264 4598 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:42.705008 4569 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.808184 1702 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:42.710779 4654 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:42.803869 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 471591us
I20260927 23:31:42.784821 4264 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.791987 4593 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.806576 4594 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.784936 4389 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:42.791700 4595 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:42.830912 1702 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:42.835264 1735 raft_consensus.cc:1261] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:42.832780 1731 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:42.808264 4630 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:42.707432 4565 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.702961 4674 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.133s
I20260927 23:31:42.833292 4669 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 23:31:42.848632 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1611) took 1906 ms (client timeout 2090 ms). Trace:
W20260927 23:31:42.872424 1580 rpcz_store.cc:269] 0927 23:31:40.941781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:40.942739 (+ 958us) service_pool.cc:227] Handling call
0927 23:31:42.848402 (+1905663us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540030c854. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:42.848541 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3838913408}
I20260927 23:31:42.815896 4636 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:42.814939 4640 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.869859 4565 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.833479 4263 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.874187 1737 raft_consensus.cc:2454] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:42.840010 4593 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.842355 4389 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.842665 4595 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.876827 1730 raft_consensus.cc:1261] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 23:31:42.858448 4594 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:42.879528 4586 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.880194 4310 raft_consensus.cc:3032] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:42.882424 4622 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:42.863319 4630 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.826460 4301 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.883746 4485 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:42.894940 4496 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.897161 4310 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:42.902891 4595 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.903322 4310 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.885387 1736 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:42.889518 4597 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:42.813694 4598 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.887625 4637 leader_election.cc:288] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.899080 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:42.837996 4264 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.890910 4600 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.915186 4264 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.905258 4397 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:42.893134 4646 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:42.914613 1736 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:42.886348 4636 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:42.924356 1658 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
W20260927 23:31:42.896633 1700 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.032s sys 0.037s
I20260927 23:31:42.902068 4496 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:42.923553 1654 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.001s sys 0.015s
I20260927 23:31:42.921072 4301 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:42.930207 1733 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.004s sys 0.012s
W20260927 23:31:42.930702 1733 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.005s sys 0.012s
W20260927 23:31:42.928453 1654 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.534s user 0.001s sys 0.015s
W20260927 23:31:42.926726 1700 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.032s sys 0.037s
W20260927 23:31:42.944697 1734 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.003s sys 0.067s
W20260927 23:31:42.945116 1734 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.515s user 0.003s sys 0.067s
I20260927 23:31:42.927870 4502 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:42.957398 1657 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:42.966598 1663 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260927 23:31:42.950438 2286 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.003s sys 0.016s
I20260927 23:31:42.975718 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.967289 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:42.942776 4209 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:42.983345 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.984833 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:42.988473 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.990236 1655 raft_consensus.cc:1226] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:42.989938 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 563509us
I20260927 23:31:42.988827 1662 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260927 23:31:42.992805 1660 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.006s sys 0.019s
I20260927 23:31:42.950356 4557 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:42.957259 4594 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:42.997399 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:42.995096 1660 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.533s user 0.007s sys 0.019s
W20260927 23:31:42.975193 2286 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.563s user 0.004s sys 0.016s
I20260927 23:31:42.988715 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:43.000841 1729 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.020s sys 0.006s
I20260927 23:31:42.957441 4640 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:43.004245 1729 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.021s sys 0.006s
W20260927 23:31:42.998015 1627 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.016s sys 0.010s
W20260927 23:31:43.005676 1627 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.016s sys 0.010s
I20260927 23:31:43.005752 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.006090 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:43.006385 2292 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.002s sys 0.026s
W20260927 23:31:43.006696 2292 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s user 0.002s sys 0.026s
I20260927 23:31:43.008087 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:43.010339 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 623745us
I20260927 23:31:43.021955 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.024245 2293 raft_consensus.cc:1226] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:43.023910 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:42.977649 4502 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.022094 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.031541 1655 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:43.026654 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:43.007239 2257 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.000s sys 0.021s
W20260927 23:31:43.034348 2257 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.538s user 0.001s sys 0.021s
W20260927 23:31:43.034772 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 538806us
W20260927 23:31:43.039672 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.005s sys 0.019s
I20260927 23:31:43.040139 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:43.050123 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.005s sys 0.019s
I20260927 23:31:43.035892 4264 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260927 23:31:43.041464 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.052398 1736 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:43.054389 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:42.988310 4611 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260927 23:31:42.983232 4670 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260927 23:31:43.048943 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:42.983374 4659 consensus_queue.cc:1035] T e382c82189e743b4ace885d63e5e9560 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20260927 23:31:43.049584 1554 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.024s sys 0.053s
I20260927 23:31:43.000512 4641 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.050936 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:42.988456 4400 consensus_queue.cc:1046] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260927 23:31:43.004074 4628 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.058440 1700 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:43.045218 2257 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:43.062397 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.084887 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.068918 4326 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:43.072726 1554 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.665s user 0.024s sys 0.053s
I20260927 23:31:43.078483 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.024879 4557 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:42.998895 4209 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.044524 4589 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.074041 4389 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.010466 4594 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.027661 4668 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260927 23:31:43.056461 4687 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.078042 1552 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:43.132457 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 54499.8us
W20260927 23:31:43.104351 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 1.15091e+06us
I20260927 23:31:43.021054 4640 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.105304 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.103799 1580 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:43.038684 4502 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.111493 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.112377 1627 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:43.060319 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 413058us
I20260927 23:31:43.110720 4610 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:43.136984 1552 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.117617 4612 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.118755 4637 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.169906 4612 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.170080 4637 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.171849 4637 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.172606 4263 raft_consensus.cc:2781] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:43.104952 4600 consensus_queue.cc:1035] T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260927 23:31:43.123389 4619 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.124058 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:43.084270 4641 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.126664 4569 raft_consensus.cc:2781] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:43.080506 4586 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:43.082103 4593 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.135629 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.091063 4630 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.079897 4444 consensus_queue.cc:1035] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260927 23:31:43.138342 1656 raft_consensus.cc:2454] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 3.
I20260927 23:31:43.154644 1580 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:43.155015 1584 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:43.156000 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.157395 1583 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:43.158355 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.159224 1582 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:43.160317 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.197480 1585 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:43.161544 4496 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.166435 4610 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.166611 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.167716 4502 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.167582 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.196244 1581 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:43.162204 1551 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:43.168850 4689 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:43.089454 4628 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.113276 4598 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.113581 4589 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.183862 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.129493 4636 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.107717 4688 consensus_queue.cc:1046] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260927 23:31:43.166133 1732 raft_consensus.cc:1261] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:43.147821 4646 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260927 23:31:43.188171 4619 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.143386 4687 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.255805 1730 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.248592 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.248682 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.271636 2287 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:43.199040 4637 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:43.203284 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.145848 4656 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:43.127166 4389 consensus_queue.cc:1046] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260927 23:31:43.208091 4706 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.201206 4569 raft_consensus.cc:686] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:43.209493 4693 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.209403 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.282940 1729 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:43.214701 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.215093 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.283469 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.226698 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:43.182631 4586 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:43.227334 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:43.228183 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.171859 4612 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.227595 2292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 510) took 1436 ms. Trace:
I20260927 23:31:43.235077 1655 raft_consensus.cc:2379] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 9: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:43.232373 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:43.136271 4682 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.034s
I20260927 23:31:43.107232 4326 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.244181 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:43.275079 1728 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:43.276320 1657 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:43.208792 4705 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.206202 4689 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:43.230189 4685 log.cc:924] Time spent T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.258s user 0.000s sys 0.001s
I20260927 23:31:43.357174 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.214141 4641 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.281471 1703 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:43.216431 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:43.379720 1703 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 98359us
I20260927 23:31:43.380110 1582 raft_consensus.cc:2396] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260927 23:31:43.255110 4589 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.381134 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.183883 4654 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.287210 2288 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260927 23:31:43.311408 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.330405 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.384025 2288 pending_rounds.cc:76] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting all ops after (but not including) 8
I20260927 23:31:43.384378 2288 pending_rounds.cc:97] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260927 23:31:43.333562 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.384716 2288 raft_consensus.cc:2866] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:43.381852 1582 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:43.330148 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.249497 4708 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
W20260927 23:31:43.329558 4682 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.735s user 0.000s sys 0.038s
I20260927 23:31:43.351133 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.250643 4640 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.426008 1733 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:43.427320 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.243189 4557 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.242003 4594 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.255458 4628 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.244374 4610 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.246831 4695 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:43.441206 4695 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.441764 4654 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.356053 4669 raft_consensus.cc:686] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:43.249046 4209 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.356168 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:43.282812 4572 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s user 0.005s sys 0.012s
I20260927 23:31:43.249527 4678 raft_consensus.cc:3032] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:43.249411 4636 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:43.372269 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.452057 1661 raft_consensus.cc:2396] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:43.383116 4500 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:43.378067 1730 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:43.244253 4677 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:43.285523 2292 rpcz_store.cc:276] 0927 23:31:41.791113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:41.791310 (+ 197us) service_pool.cc:227] Handling call
0927 23:31:41.821878 (+ 30568us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:42.420084 (+598206us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:42.420111 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:42.428454 (+ 8343us) log.cc:842] Serialized 26 byte log entry
0927 23:31:43.007031 (+578577us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:43.007070 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.037426 (+ 30356us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540059f454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:43.037502 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:43.215762 (+178260us) raft_consensus.cc:1671] finished
0927 23:31:43.215787 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:43.227514 (+ 11727us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43147,"raft.queue_time_us":7154,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11754,"spinlock_wait_cycles":1358633728,"thread_start_us":518538,"threads_started":1,"wal-append.queue_time_us":752125}
I20260927 23:31:43.282166 4301 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.253719 4310 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.385917 1655 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:43.462486 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.463454 1588 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:43.384503 1732 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:43.310563 4586 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:43.396227 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1902) took 1057 ms. Trace:
I20260927 23:31:43.342543 4656 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:43.356593 4689 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:43.436558 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.436359 4708 raft_consensus.cc:686] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:43.439635 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.445343 2288 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 513) took 1101 ms. Trace:
I20260927 23:31:43.443301 4695 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.357954 1729 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:43.384934 1728 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.426838 4612 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.464923 1654 rpcz_store.cc:276] 0927 23:31:42.338683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.338798 (+ 115us) service_pool.cc:227] Handling call
0927 23:31:42.339086 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.339344 (+ 258us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:42.942748 (+603404us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.395814 (+453066us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.395878 (+ 64us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:43.395944 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.396025 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:43.396162 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13400,"prepare.queue_time_us":819039,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":55010,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":5440,"thread_start_us":979068,"threads_started":2}
I20260927 23:31:43.329573 4693 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.310536 4706 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.378979 4705 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.473644 1663 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
W20260927 23:31:43.541786 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 483413us
I20260927 23:31:43.460201 4677 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:43.546718 1735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.016s sys 0.026s
W20260927 23:31:43.561164 1735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.017s sys 0.026s
I20260927 23:31:43.561246 1700 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.484623 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.495325 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.458307 4678 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:43.546432 1660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.007s sys 0.031s
W20260927 23:31:43.563239 1660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.008s sys 0.031s
I20260927 23:31:43.563388 1582 raft_consensus.cc:2396] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:43.514639 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:43.564023 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 403) took 1102 ms. Trace:
I20260927 23:31:43.483362 2288 rpcz_store.cc:276] 0927 23:31:42.344442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.385047 (+ 40605us) service_pool.cc:227] Handling call
0927 23:31:42.385322 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:43.385294 (+999972us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.445139 (+ 59845us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54006dea54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:43.445293 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2073171712}
I20260927 23:31:43.557716 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.495599 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.513554 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:43.522890 1659 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.042s sys 0.039s
W20260927 23:31:43.574008 1659 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s user 0.042s sys 0.042s
I20260927 23:31:43.574781 4585 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 23:31:43.574906 1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 404) took 1113 ms. Trace:
I20260927 23:31:43.575497 2259 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:43.576364 4400 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:43.576699 4400 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
W20260927 23:31:43.541499 4622 consensus_meta.cc:287] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.635s user 0.009s sys 0.001s
I20260927 23:31:43.562424 1661 raft_consensus.cc:2379] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:43.566849 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.591857 1732 raft_consensus.cc:2379] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:43.490613 4569 consensus_queue.cc:237] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.569849 4670 consensus_queue.cc:1035] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260927 23:31:43.577720 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.577654 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.579638 1728 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:43.580319 1700 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.580924 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:43.593132 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.597234 1587 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:43.483652 1733 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:43.458253 4500 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:43.575419 1659 rpcz_store.cc:276] 0927 23:31:42.461829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.464332 (+ 2503us) service_pool.cc:227] Handling call
0927 23:31:42.486094 (+ 21762us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.492584 (+ 6490us) raft_consensus.cc:1520] Early marking committed up to index 9
0927 23:31:42.950025 (+457441us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.574472 (+624447us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.574539 (+ 67us) raft_consensus.cc:1631] Marking committed up to 9
0927 23:31:43.574602 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.574695 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:43.574847 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":676728,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":1710,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"thread_start_us":1074291,"threads_started":2}
I20260927 23:31:43.443318 4669 consensus_queue.cc:237] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.600853 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:43.602617 1702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.001s sys 0.028s
I20260927 23:31:43.564435 1660 rpcz_store.cc:276] 0927 23:31:42.461560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.461670 (+ 110us) service_pool.cc:227] Handling call
0927 23:31:42.461988 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.462175 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 9
0927 23:31:43.002962 (+540787us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.563611 (+560649us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.563679 (+ 68us) raft_consensus.cc:1631] Marking committed up to 9
0927 23:31:43.563746 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.563827 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:43.563961 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":706989,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":9084,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":1100662,"threads_started":2}
W20260927 23:31:43.605665 1702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.001s sys 0.028s
I20260927 23:31:43.511286 4698 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:43.577024 4400 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:43.462484 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 417299us
W20260927 23:31:43.608116 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 445948us
I20260927 23:31:43.618924 1584 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:43.620282 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.621662 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:43.621762 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:43.622396 1584 raft_consensus.cc:595] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:43.622933 1584 raft_consensus.cc:603] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:43.624994 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.625427 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.626036 1654 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:43.629441 4565 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.629627 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.630002 4565 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.631527 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.631290 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.632565 4565 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.632908 1654 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:43.633338 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.633935 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:43.636442 1654 raft_consensus.cc:2379] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:43.625285 1733 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:43.639531 4397 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.640079 4397 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.642638 4397 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:43.643621 1658 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.010s sys 0.014s
W20260927 23:31:43.644088 1658 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.504s user 0.010s sys 0.014s
I20260927 23:31:43.461699 4636 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.655992 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.657661 4628 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:43.658304 4628 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:43.659395 4628 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.583662 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 565) took 1177 ms. Trace:
I20260927 23:31:43.465610 4301 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.606676 4500 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:43.591204 4715 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:43.677556 1588 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:43.684165 1662 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:43.691963 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.693264 1584 raft_consensus.cc:2379] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:43.694489 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.648983 4572 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:43.560840 4612 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.697135 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.711735 1585 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.712327 1654 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:43.697973 1551 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:43.698383 1736 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.000s sys 0.017s
W20260927 23:31:43.713168 1736 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.586s user 0.001s sys 0.017s
I20260927 23:31:43.701371 1627 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:43.600540 4695 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260927 23:31:43.715737 1656 raft_consensus.cc:2396] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260927 23:31:43.651145 4619 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.717146 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.721446 4397 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:43.721640 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.722568 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:43.728026 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.733172 1654 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:43.735195 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.673260 4687 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.720631 1584 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
I20260927 23:31:43.743278 1702 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.688594 4326 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.719512 1587 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:43.769345 1585 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:43.729485 4397 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:43.825434 4397 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.733621 4686 raft_consensus.cc:3032] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:43.826491 4686 raft_consensus.cc:729] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:43.827286 4686 consensus_queue.cc:260] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.828344 4686 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.829267 4400 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.829835 4693 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:43.745426 4709 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
I20260927 23:31:43.830826 2294 raft_consensus.cc:1226] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:43.740248 4598 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:43.831282 4738 consensus_queue.cc:1046] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:43.831941 4301 leader_election.cc:288] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.609582 4656 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.832459 4301 raft_consensus.cc:3032] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:43.832770 4301 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:43.833493 4301 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.665778 4698 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.794984 4612 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:43.842020 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:43.851794 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.852625 1657 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
W20260927 23:31:43.852789 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 236967us
I20260927 23:31:43.697642 4708 consensus_queue.cc:237] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.715833 4678 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:43.814890 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:43.816082 4714 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
I20260927 23:31:43.858438 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:43.742385 4263 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.019s sys 0.014s
W20260927 23:31:43.817684 4719 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260927 23:31:43.818939 4706 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.819341 4595 raft_consensus.cc:3032] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Stepping down as leader of term 8
W20260927 23:31:43.820417 4718 log.cc:924] Time spent T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 23:31:43.901528 1656 raft_consensus.cc:1226] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260927 23:31:43.831498 4598 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.782936 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:43.835129 1588 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
W20260927 23:31:43.837355 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 94169.2us
I20260927 23:31:43.926762 1702 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.928571 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:43.841935 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:43.896634 4605 consensus_meta.cc:287] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 1.184s user 0.006s sys 0.006s
I20260927 23:31:43.856259 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.930473 4598 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:43.941227 4689 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:43.862286 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 243222us
I20260927 23:31:43.859133 1657 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:43.866146 4301 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:43.871403 1552 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.952078 4646 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:43.952495 4646 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:43.953231 4646 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:43.831316 4400 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:43.696805 4677 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.957203 4677 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:43.959491 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 88148.5us
I20260927 23:31:43.901630 4326 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:43.960238 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.960125 4326 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:43.961020 4733 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:43.961311 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.961653 4733 raft_consensus.cc:686] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:43.962212 4731 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:43.835736 4737 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:43.729758 4485 consensus_queue.cc:1035] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260927 23:31:43.897501 4595 raft_consensus.cc:729] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:43.855463 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:43.925549 1654 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
W20260927 23:31:43.883193 4263 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.020s sys 0.014s
I20260927 23:31:43.931413 1702 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.859524 4716 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:43.861460 4619 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:43.863085 4565 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:43.941411 4725 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:43.963631 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.936702 4678 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:43.941520 4687 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:43.965652 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:43.962060 4326 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:43.902909 1628 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:43.893306 4695 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260927 23:31:43.986528 4737 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:43.963888 1737 raft_consensus.cc:2354] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:43.897236 4682 consensus_queue.cc:1046] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260927 23:31:43.900504 4693 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
W20260927 23:31:44.011009 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 79693.7us
I20260927 23:31:44.033319 1702 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:43.856799 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 232118us
W20260927 23:31:44.014055 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 111250us
I20260927 23:31:44.036093 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:44.037098 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.005769 1552 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.037882 1730 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:43.990377 4263 raft_consensus.cc:686] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:43.988830 4726 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.039019 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:43.967624 4731 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:43.990748 4557 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:43.991920 4705 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.011955 4706 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:44.012161 4716 raft_consensus.cc:686] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:44.015151 4589 raft_consensus.cc:2726] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:44.020025 4612 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:44.054534 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.034749 4737 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.049393 4725 raft_consensus.cc:2726] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:44.063030 4728 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:44.008284 4619 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.042431 4326 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:44.046758 4636 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.044960 4689 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:44.072330 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.012200 4565 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.086968 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.089916 1581 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:44.091405 4656 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.092438 4656 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:44.096031 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57151.9us
I20260927 23:31:44.096845 4656 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.096997 1552 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.097945 4656 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
W20260927 23:31:44.099932 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 63958.2us
I20260927 23:31:44.100586 1628 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.100621 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.102066 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.057047 4706 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:44.104462 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.106338 2288 raft_consensus.cc:1261] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 23:31:44.108006 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.109923 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:43.668681 2286 rpcz_store.cc:276] 0927 23:31:42.406544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.411791 (+ 5247us) service_pool.cc:227] Handling call
0927 23:31:42.412056 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.412250 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 6
0927 23:31:43.027915 (+615665us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.536971 (+509056us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.537057 (+ 86us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:43.537158 (+ 101us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.542690 (+ 5532us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:43.563091 (+ 20401us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":756243,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":4033,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4033,"thread_start_us":1064267,"threads_started":2,"wal-append.queue_time_us":783957}
I20260927 23:31:44.119262 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.148169 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.152450 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:44.108611 1552 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.066388 4728 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:44.005831 4733 consensus_queue.cc:237] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.157255 1736 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.048123 4726 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.165307 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:44.165684 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 57153us
I20260927 23:31:44.167474 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.102306 4636 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:44.054538 4557 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.171372 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfb0bd166774a248820c1b90412697d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.104506 4326 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:44.172923 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:44.173210 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 140031us
I20260927 23:31:44.120574 4706 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.173756 4695 raft_consensus.cc:2726] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260927 23:31:44.192858 4740 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260927 23:31:44.103523 4656 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.149593 4698 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.181877 1702 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.148447 4612 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:44.207243 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:44.180200 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 79680.6us
W20260927 23:31:44.215313 4752 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20260927 23:31:44.224143 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:44.224979 2292 raft_consensus.cc:2396] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260927 23:31:44.226444 1736 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:44.163138 4687 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:44.231716 1700 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.172701 4732 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:44.232334 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:44.240893 4310 consensus_meta.cc:287] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.773s user 0.011s sys 0.006s
I20260927 23:31:44.243981 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.070978 4595 consensus_queue.cc:260] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.243132 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.260597 1586 raft_consensus.cc:2396] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:44.228057 4716 consensus_queue.cc:237] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.245047 4687 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.274437 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.183960 4636 raft_consensus.cc:686] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:44.263778 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.262840 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.274578 1736 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:44.271725 4726 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:44.283833 2288 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
W20260927 23:31:44.262887 4751 log.cc:924] Time spent T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260927 23:31:44.318534 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.315140 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.321885 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.322597 1736 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 4.
I20260927 23:31:44.324107 2258 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:44.325680 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:44.309293 4742 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.367s user 0.000s sys 0.003s
I20260927 23:31:44.329752 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.165326 4263 consensus_queue.cc:237] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:44.326964 4749 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260927 23:31:44.316989 4595 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:44.335817 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.337189 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 469) took 1128 ms. Trace:
I20260927 23:31:44.317234 4326 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.339250 1732 rpcz_store.cc:276] 0927 23:31:43.208686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.592814 (+384128us) service_pool.cc:227] Handling call
0927 23:31:43.609341 (+ 16527us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:43.619621 (+ 10280us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.619661 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:43.650517 (+ 30856us) log.cc:842] Serialized 26 byte log entry
0927 23:31:43.968005 (+317488us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:43.968042 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.968104 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:44.336970 (+368866us) raft_consensus.cc:1671] finished
0927 23:31:44.337007 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:44.337131 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9777,"raft.queue_time_us":9885,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":22026,"thread_start_us":252991,"threads_started":1,"wal-append.queue_time_us":389818}
I20260927 23:31:44.330108 4726 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:44.175328 4731 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.330060 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.325783 4766 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.336169 4640 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:44.358691 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.345758 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.354761 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:44.355436 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1683) took 1387 ms. Trace:
I20260927 23:31:44.350031 1554 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.248005 4728 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.355950 4557 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.360494 1655 rpcz_store.cc:276] 0927 23:31:42.968449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.469898 (+501449us) service_pool.cc:227] Handling call
0927 23:31:43.470307 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:43.470539 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.470575 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:43.471041 (+ 466us) log.cc:842] Serialized 26 byte log entry
0927 23:31:43.761344 (+290303us) raft_consensus.cc:1631] Marking committed up to 3
0927 23:31:43.788471 (+ 27127us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.788576 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:44.355218 (+566642us) raft_consensus.cc:1671] finished
0927 23:31:44.355238 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:44.355382 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":221288,"threads_started":1,"wal-append.queue_time_us":402032}
I20260927 23:31:44.344776 4726 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:44.377441 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.374825 1554 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.368474 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.344844 4565 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.380460 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.381273 1732 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:44.380251 1736 raft_consensus.cc:2379] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:44.382619 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.381937 1554 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.384052 1732 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:44.385564 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.367683 4569 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:44.271479 4682 raft_consensus.cc:2932] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:44.392907 4569 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.395581 4569 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.397269 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.399297 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:44.401945 4572 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.687s user 0.005s sys 0.002s
I20260927 23:31:44.404248 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:44.406399 4761 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260927 23:31:44.407344 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:44.405588 4586 consensus_meta.cc:287] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.921s user 0.010s sys 0.000s
I20260927 23:31:44.409920 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:44.413278 1736 raft_consensus.cc:2454] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:44.414968 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1554) took 1208 ms. Trace:
I20260927 23:31:44.415194 4734 consensus_queue.cc:1035] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:44.415383 1582 rpcz_store.cc:276] 0927 23:31:43.206661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.647037 (+440376us) service_pool.cc:227] Handling call
0927 23:31:43.647357 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:43.647807 (+ 450us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.647844 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:43.666966 (+ 19122us) log.cc:842] Serialized 26 byte log entry
0927 23:31:44.017119 (+350153us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:44.017174 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.017500 (+ 326us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:44.414765 (+397265us) raft_consensus.cc:1671] finished
0927 23:31:44.414783 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:44.414910 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":20370,"thread_start_us":349711,"threads_started":1,"wal-append.queue_time_us":451769}
I20260927 23:31:44.416807 4565 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.417162 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.417455 1554 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.417471 4565 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.418701 1554 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.419833 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.420293 4565 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.420966 4565 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:44.421667 4565 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:44.423350 4565 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:44.424330 4565 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:44.345062 4326 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.425203 4565 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:44.426299 4565 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:44.427802 1582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:44.431211 4646 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.430827 1582 raft_consensus.cc:595] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.426582 4326 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.409976 4586 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.427452 1736 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:44.428079 4565 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:44.406968 1587 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 10.
I20260927 23:31:44.433447 1582 raft_consensus.cc:603] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:44.358917 4640 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:44.377132 4500 consensus_meta.cc:287] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.665s user 0.010s sys 0.000s
I20260927 23:31:44.362252 4619 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.413678 4706 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.438937 4572 raft_consensus.cc:2726] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:44.381916 4726 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:44.479797 1732 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:44.483523 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 63753us
I20260927 23:31:44.416489 4734 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:44.470846 4686 consensus_meta.cc:287] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.642s user 0.010s sys 0.001s
I20260927 23:31:44.477348 2259 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:44.496270 1730 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:44.496978 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:44.497548 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 255002us
I20260927 23:31:44.497759 4659 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260927 23:31:44.498246 4659 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:44.498590 4659 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:44.499029 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:44.471678 4740 log.cc:924] Time spent T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
W20260927 23:31:44.475714 4677 consensus_meta.cc:287] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.518s user 0.004s sys 0.005s
I20260927 23:31:44.481472 4640 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:44.489665 1579 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:44.492486 1729 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260927 23:31:44.493927 4734 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.494774 4706 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.480492 4565 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:44.506572 1628 leader_election.cc:302] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:44.506788 1736 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
W20260927 23:31:44.511415 4711 log.cc:924] Time spent T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.910s user 0.002s sys 0.002s
I20260927 23:31:44.501219 4680 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:44.487869 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.377748 4636 consensus_queue.cc:237] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.365618 4766 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.547405 1628 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.548388 1582 raft_consensus.cc:1261] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:44.471118 4598 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.550406 4619 raft_consensus.cc:2726] T 4cfb0bd166774a248820c1b90412697d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:44.557276 4677 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:44.587950 1587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:44.588688 1587 raft_consensus.cc:595] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:44.551836 4680 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:44.560127 1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 456) took 1718 ms. Trace:
I20260927 23:31:44.547160 4656 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.487510 4500 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.604295 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 23:31:44.599078 4677 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:44.594990 1731 rpcz_store.cc:276] 0927 23:31:42.841615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.841762 (+ 147us) service_pool.cc:227] Handling call
0927 23:31:42.871350 (+ 29588us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 23:31:43.356396 (+485046us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.356426 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 23:31:43.395614 (+ 39188us) log.cc:842] Serialized 52 byte log entry
0927 23:31:43.750088 (+354474us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:43.786144 (+ 36056us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.818517 (+ 32373us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.818715 (+ 198us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.818782 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:44.559898 (+741116us) raft_consensus.cc:1671] finished
0927 23:31:44.559927 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:44.560068 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43242,"prepare.queue_time_us":38845,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":75,"raft.queue_time_us":85554,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":7483,"thread_start_us":771446,"threads_started":2,"wal-append.queue_time_us":500690}
W20260927 23:31:44.624498 4678 consensus_meta.cc:287] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.612s user 0.007s sys 0.003s
I20260927 23:31:44.626634 1628 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:44.636346 1584 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:44.630764 2288 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:44.628224 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 23:31:44.640270 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1550) took 1479 ms. Trace:
I20260927 23:31:44.641795 1585 rpcz_store.cc:276] 0927 23:31:43.160951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.198141 (+ 37190us) service_pool.cc:227] Handling call
0927 23:31:44.635967 (+1437826us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54003b3dd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:44.639981 (+ 4014us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":982356224}
I20260927 23:31:44.650422 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:44.653415 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:44.587634 4737 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.655827 4737 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.659633 4500 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:44.661224 4397 consensus_meta.cc:287] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.835s user 0.011s sys 0.000s
I20260927 23:31:44.624811 4680 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.661623 4397 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.662225 4680 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:44.663573 4397 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.664249 1735 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 23:31:44.670508 4557 consensus_queue.cc:1046] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:44.659628 1731 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260927 23:31:44.671609 4557 raft_consensus.cc:3032] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:44.671937 4557 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:44.672554 4612 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:44.660749 4726 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.666283 1735 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:44.696991 4598 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.698536 4776 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.699147 4776 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.700279 4766 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.700893 4766 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.701586 4776 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.701375 4766 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:44.660804 4301 consensus_meta.cc:287] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.712s user 0.006s sys 0.009s
I20260927 23:31:44.639303 4619 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 23:31:44.707391 4779 log.cc:924] Time spent T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260927 23:31:44.709332 4734 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:44.704782 4753 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260927 23:31:44.631166 4677 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.737128 1735 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:44.739684 1735 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:44.740612 1731 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:44.749610 4770 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.754329 4768 raft_consensus.cc:3032] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:44.694748 4397 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.755451 4744 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:44.757527 4598 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:44.760390 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 470) took 1523 ms. Trace:
I20260927 23:31:44.760984 4766 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.769903 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:44.776597 1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46518 (request call id 1555) took 1548 ms (client timeout 1824 ms). Trace:
W20260927 23:31:44.780020 1586 rpcz_store.cc:269] 0927 23:31:43.229724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.243946 (+ 14222us) service_pool.cc:227] Handling call
0927 23:31:44.776308 (+1532362us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b5400061154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:44.776526 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2102990720}
I20260927 23:31:44.631692 4586 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.778020 1728 rpcz_store.cc:276] 0927 23:31:43.236748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.603156 (+366408us) service_pool.cc:227] Handling call
0927 23:31:43.603517 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:43.603925 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.603951 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:43.645408 (+ 41457us) log.cc:842] Serialized 26 byte log entry
0927 23:31:44.008382 (+362974us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:44.008421 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.008489 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:44.495668 (+487179us) spinlock_profiling.cc:243] Waited 5.9 ms on lock 0x7b4800250158. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97559167f1 00007f975590d8e7 00007f975590d77a 00007f9755a0195f 00007f9755a01627 00007f9755a15b2c 00007f9755a15ad8 00007f9755a15a58 00007f9755a15a13 00007f9755a14cbc 00007f9754996be6
0927 23:31:44.760111 (+264443us) raft_consensus.cc:1671] finished
0927 23:31:44.760131 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:44.760329 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11479,"raft.run_cpu_time_us":7459,"raft.run_wall_time_us":850887,"spinlock_wait_cycles":13562112,"thread_start_us":297108,"threads_started":1,"wal-append.queue_time_us":952173}
I20260927 23:31:44.803440 1731 raft_consensus.cc:1226] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:44.804224 1655 raft_consensus.cc:1226] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:44.805958 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:44.672605 4557 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.807915 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:44.808604 1657 raft_consensus.cc:2362] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5. Re-sending same reply.
I20260927 23:31:44.809531 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:44.810055 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:44.810952 1657 raft_consensus.cc:2362] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8. Re-sending same reply.
I20260927 23:31:44.816247 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:44.816308 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:44.818006 1586 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:44.819876 1654 raft_consensus.cc:1261] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 23:31:44.821399 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:44.822114 1660 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:44.767234 4706 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:44.777997 4768 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:44.777772 4744 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
W20260927 23:31:44.825220 4732 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.001s sys 0.011s
I20260927 23:31:44.857754 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:44.861079 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:44.811445 4775 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:44.791747 4659 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.823527 4726 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.792502 4397 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:44.861917 4732 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.001s sys 0.011s
I20260927 23:31:44.878844 1586 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 23:31:44.892438 1586 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:44.873039 1581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1546) took 1713 ms. Trace:
I20260927 23:31:44.893712 1581 rpcz_store.cc:276] 0927 23:31:43.159883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.217571 (+ 57688us) service_pool.cc:227] Handling call
0927 23:31:43.217887 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.090489 (+872602us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.872823 (+782334us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b540007d354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:44.872986 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2004930816}
I20260927 23:31:44.896580 2294 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 523) took 1292 ms. Trace:
I20260927 23:31:44.812326 4640 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.890209 4732 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:44.899773 4397 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.806587 4598 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.896935 2294 rpcz_store.cc:276] 0927 23:31:43.604590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.604819 (+ 229us) service_pool.cc:227] Handling call
0927 23:31:43.605082 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:43.831180 (+226098us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.896373 (+1065193us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540059f454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:44.896528 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":518739200}
I20260927 23:31:44.911111 4768 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.919153 4728 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.881796 4775 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.919440 4768 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:44.920094 4744 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.792693 4770 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.921232 4744 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:44.828310 4706 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.922585 4770 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:44.923977 4706 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:44.919409 4787 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.828492 4565 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:44.924906 4787 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.899893 4732 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:44.927067 1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46522 (request call id 1645) took 1185 ms. Trace:
I20260927 23:31:44.935145 1584 rpcz_store.cc:276] 0927 23:31:43.741463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.741824 (+ 361us) service_pool.cc:227] Handling call
0927 23:31:44.636115 (+894291us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.636842 (+ 727us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.926860 (+290018us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54003b3dd0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:44.926999 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2055821184}
I20260927 23:31:44.885366 4766 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:44.943732 1583 raft_consensus.cc:1226] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:44.971243 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 112018us
W20260927 23:31:44.953373 4682 consensus_meta.cc:287] T 9c719671f35640549c1b944112b3fc03 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.560s user 0.011s sys 0.000s
I20260927 23:31:44.944756 4766 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } attrs { replace: true } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.232715 1660 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:45.233407 1581 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:45.234279 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:45.059337 1582 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 504) took 1166 ms. Trace:
I20260927 23:31:45.235008 2294 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:45.239356 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.055552 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:45.253443 1584 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:45.251852 1586 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:45.251488 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48306 (request call id 521) took 1710 ms. Trace:
I20260927 23:31:45.258193 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1755) took 2832 ms. Trace:
W20260927 23:31:45.042255 4760 log.cc:924] Time spent T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260927 23:31:45.258077 2293 rpcz_store.cc:276] 0927 23:31:43.541536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.541672 (+ 136us) service_pool.cc:227] Handling call
0927 23:31:43.592489 (+ 50817us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:43.864399 (+271910us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.864441 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.864492 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0927 23:31:43.864535 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:45.251281 (+1386746us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b540071f2d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:45.251338 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:45.251439 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":624824576}
I20260927 23:31:45.000455 4565 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:45.248492 1582 rpcz_store.cc:276] 0927 23:31:43.893331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.631498 (+738167us) service_pool.cc:227] Handling call
0927 23:31:44.631959 (+ 461us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.632360 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 23:31:44.941949 (+309589us) log.cc:842] Serialized 14 byte log entry
0927 23:31:45.058902 (+116953us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:45.058970 (+ 68us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:45.059034 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:45.059111 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:45.059267 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":419065,"raft.queue_time_us":86316,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":199446,"threads_started":2}
I20260927 23:31:45.262742 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.157296 4611 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:45.000586 4737 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.269307 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:45.258970 1734 rpcz_store.cc:276] 0927 23:31:42.418225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.418401 (+ 176us) service_pool.cc:227] Handling call
0927 23:31:42.418718 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:42.429939 (+ 11221us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 23:31:42.945502 (+515563us) log.cc:842] Serialized 14 byte log entry
0927 23:31:43.461283 (+515781us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:43.461331 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:43.461375 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:43.883675 (+422300us) spinlock_profiling.cc:243] Waited 6.28 ms on lock 0x7b18003403c0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97569e68b1 00007f97569f3661 00007f974db14f96 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9
0927 23:31:45.249989 (+1366314us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b54003c4454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:45.250051 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:45.250194 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74524,"prepare.queue_time_us":743834,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":10342,"raft.queue_time_us":74653,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6793,"spinlock_wait_cycles":39633536,"thread_start_us":956072,"threads_started":2,"wal-append.queue_time_us":699585,"wal-append.run_cpu_time_us":9356,"wal-append.run_wall_time_us":1320111}
I20260927 23:31:45.273185 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.234475 4738 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:45.045636 4775 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.043140 4787 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.045289 4659 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:45.234524 4637 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:45.278033 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1771) took 1437 ms. Trace:
I20260927 23:31:45.278414 2294 raft_consensus.cc:2454] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:45.290614 1554 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:45.299041 1584 raft_consensus.cc:2454] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 11.
I20260927 23:31:45.284617 1733 rpcz_store.cc:276] 0927 23:31:43.840944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.841053 (+ 109us) service_pool.cc:227] Handling call
0927 23:31:43.841337 (+ 284us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:43.841633 (+ 296us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:43.841659 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:43.842019 (+ 360us) log.cc:842] Serialized 26 byte log entry
0927 23:31:44.111686 (+269667us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:44.169675 (+ 57989us) log.cc:842] Serialized 14 byte log entry
0927 23:31:44.169908 (+ 233us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.169991 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:45.277843 (+1107852us) raft_consensus.cc:1671] finished
0927 23:31:45.277864 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:45.277979 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31074,"prepare.queue_time_us":55597,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2757,"raft.queue_time_us":2760,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8651,"spinlock_wait_cycles":14771712,"thread_start_us":269156,"threads_started":1,"wal-append.queue_time_us":960835}
I20260927 23:31:45.255782 4677 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.283587 4768 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:45.276894 4611 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.283324 4737 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.287616 4738 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.287314 4637 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.559803 2292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 572) took 1455 ms. Trace:
I20260927 23:31:45.714751 2292 rpcz_store.cc:276] 0927 23:31:44.105207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.107673 (+ 2466us) service_pool.cc:227] Handling call
0927 23:31:45.530503 (+1422830us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b54006dea54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:45.559711 (+ 29208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46804352}
W20260927 23:31:45.653965 4744 consensus_meta.cc:287] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.732s user 0.009s sys 0.004s
I20260927 23:31:45.465062 4787 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:45.560663 4598 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:45.530005 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 59990.7us
W20260927 23:31:45.731251 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 327739us
I20260927 23:31:45.566237 2294 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260927 23:31:45.723141 4565 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.740422 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:45.741822 1584 raft_consensus.cc:2396] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:45.316584 4733 raft_consensus.cc:2781] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:45.747331 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:45.763759 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 497135us
I20260927 23:31:45.727885 4677 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.768333 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:45.732597 4775 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.329986 4640 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.772909 1586 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:45.768267 1554 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:45.785104 2288 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48296 (request call id 583) took 1288 ms (client timeout 1650 ms). Trace:
I20260927 23:31:45.786424 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:45.785743 2288 rpcz_store.cc:269] 0927 23:31:44.496623 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.496761 (+ 138us) service_pool.cc:227] Handling call
0927 23:31:45.784882 (+1288121us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54005de154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:45.785028 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306184960}
I20260927 23:31:45.729242 4787 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.794593 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:45.787480 4677 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.776913 4733 raft_consensus.cc:686] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:45.742591 4768 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.766826 4565 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.788007 4598 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 23:31:45.782401 4775 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:45.792480 4682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.014s sys 0.019s
I20260927 23:31:45.806115 4738 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.809530 4637 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:45.823132 4682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.014s sys 0.020s
I20260927 23:31:45.835187 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:45.837616 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.838286 1586 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:45.825302 4738 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:45.833297 4637 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:45.832588 4766 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:45.846769 1629 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:45.847213 2292 raft_consensus.cc:1226] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:45.836552 4677 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.860944 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:45.861445 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.866063 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.866748 1586 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:45.873212 4640 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.873780 4640 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.874701 4807 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:45.874989 4811 consensus_queue.cc:1046] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 23:31:45.876744 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.866182 4677 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.863327 4737 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.843417 4611 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:45.883196 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 122534us
W20260927 23:31:45.886942 4801 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 23:31:45.888530 4775 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.875979 4811 raft_consensus.cc:3032] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:45.889228 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.889353 4811 raft_consensus.cc:729] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:45.890836 4565 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.890465 4811 consensus_queue.cc:260] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.891677 4811 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:45.893808 4640 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.842979 4738 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.894692 1586 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:45.897563 4737 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 23:31:45.898206 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:45.901149 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:45.881119 4796 log.cc:924] Time spent T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260927 23:31:45.902161 1584 raft_consensus.cc:595] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:45.904482 1584 raft_consensus.cc:603] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 23:31:45.902139 4798 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260927 23:31:45.907851 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.842149 4733 consensus_queue.cc:237] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.847469 4637 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:45.910879 4737 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:45.911490 4737 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:45.913106 4738 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.913434 4737 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.909507 4787 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:45.915782 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:45.916381 1584 raft_consensus.cc:595] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:45.916553 1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38820 (request call id 1770) took 2075 ms (client timeout 2112 ms). Trace:
I20260927 23:31:45.916754 1584 raft_consensus.cc:603] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 23:31:45.916972 1737 rpcz_store.cc:269] 0927 23:31:43.840665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.841801 (+ 1136us) service_pool.cc:227] Handling call
0927 23:31:45.916396 (+2074595us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002ee0d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:45.916506 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257203072}
I20260927 23:31:45.917402 4637 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:45.914141 4804 log.cc:924] Time spent T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260927 23:31:45.902535 4768 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:45.924633 2259 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:45.931102 1582 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:45.933503 1582 raft_consensus.cc:595] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:45.933939 1582 raft_consensus.cc:603] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:45.934975 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.931337 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:45.931142 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.937794 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.932132 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:45.946170 1586 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 5.
I20260927 23:31:45.948154 1655 raft_consensus.cc:1226] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260927 23:31:45.948202 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.956003 1584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:45.956794 1584 raft_consensus.cc:595] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:45.962553 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 63410.2us
I20260927 23:31:45.963939 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:45.965586 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:45.966805 1554 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:45.967374 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:45.969116 4674 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:45.969523 4674 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:45.969903 4674 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:45.946298 4810 raft_consensus.cc:2781] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 FOLLOWER]: Leader election won for term 11
I20260927 23:31:45.978358 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:45.980736 1736 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:45.985419 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.986258 1655 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:45.989755 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.991341 1655 raft_consensus.cc:2379] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:45.994359 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:45.999229 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.005760 4766 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:46.006415 4766 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.008695 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.011065 1656 raft_consensus.cc:1261] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 23:31:46.011862 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.012171 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.012820 4646 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.013537 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.014109 1661 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:46.014834 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.015347 1661 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:46.017081 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.016049 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.019467 1628 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:46.019748 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.021557 4775 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:46.021915 4775 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.022233 4775 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:46.026510 4768 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:46.027876 4768 consensus_queue.cc:1046] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:46.027989 4640 raft_consensus.cc:3032] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:46.028515 4640 raft_consensus.cc:729] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.028896 4768 consensus_queue.cc:1046] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:46.030148 4737 raft_consensus.cc:3032] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:46.030527 4737 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.031199 4737 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.032269 4737 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
W20260927 23:31:46.033653 1580 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1655) took 2224 ms (client timeout 2035 ms). Trace:
W20260927 23:31:46.037907 4759 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 1.082s user 0.012s sys 0.012s
I20260927 23:31:46.042308 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.043478 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:46.044497 4784 log.cc:924] Time spent T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.082s user 0.011s sys 0.011s
W20260927 23:31:46.042302 4757 log.cc:924] Time spent T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 1.077s user 0.006s sys 0.018s
W20260927 23:31:46.046025 4712 log.cc:924] Time spent T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 2.289s user 0.003s sys 0.029s
I20260927 23:31:46.012861 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.047956 1656 raft_consensus.cc:2379] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:46.048309 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1644) took 2080 ms. Trace:
I20260927 23:31:46.048659 1732 rpcz_store.cc:276] 0927 23:31:43.967981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.409114 (+441133us) service_pool.cc:227] Handling call
0927 23:31:44.409408 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.507338 (+ 97930us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.048110 (+1540772us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b540011bf54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.048254 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161416960}
I20260927 23:31:46.049666 4715 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:46.050966 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 1.81916e+06us
I20260927 23:31:46.051595 1700 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.051661 4716 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.052495 4716 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:46.052856 4716 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.053236 4716 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.053819 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.054186 4594 raft_consensus.cc:482] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.054636 4687 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.054720 4594 raft_consensus.cc:504] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.055337 1733 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:46.055940 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 477) took 1724 ms. Trace:
W20260927 23:31:46.049829 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 1.86796e+06us
I20260927 23:31:46.056797 4594 leader_election.cc:288] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.055107 1581 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:46.056977 1702 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.057116 4557 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.052733 1700 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.058166 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.058504 4209 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:46.058583 4628 raft_consensus.cc:2726] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.059152 4209 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:46.054204 1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38812 (request call id 1645) took 2085 ms (client timeout 2000 ms). Trace:
I20260927 23:31:46.059810 1700 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:46.060314 1730 rpcz_store.cc:269] 0927 23:31:43.968314 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.120420 (+152106us) service_pool.cc:227] Handling call
0927 23:31:46.054012 (+1933592us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54001d1354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:46.054124 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":757159168}
I20260927 23:31:46.063194 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:46.063421 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.063624 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.064448 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:46.034143 1580 rpcz_store.cc:269] 0927 23:31:43.808969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.318163 (+509194us) service_pool.cc:227] Handling call
0927 23:31:46.033380 (+1715217us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003093d4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:46.033561 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787857664}
I20260927 23:31:46.059713 1702 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.044490 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.067759 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.068943 1586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:46.069634 1586 raft_consensus.cc:595] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.070762 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.068186 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.056316 1735 rpcz_store.cc:276] 0927 23:31:44.331675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.764072 (+432397us) service_pool.cc:227] Handling call
0927 23:31:44.764418 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.764641 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 23:31:44.945496 (+180855us) log.cc:842] Serialized 14 byte log entry
0927 23:31:46.055680 (+1110184us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:46.055724 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:46.055767 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.055818 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:46.055899 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":704373,"prepare.queue_time_us":862735,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":85,"raft.queue_time_us":705685,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":256607,"threads_started":2,"wal-append.queue_time_us":865401}
I20260927 23:31:46.088948 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.120992 396 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260927 23:31:46.111071 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51418.8us
I20260927 23:31:46.111270 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.008864 4766 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.091481 1700 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.122179 1737 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.122277 1702 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.018558 1661 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:46.123384 396 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260927 23:31:46.048342 4790 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 1.084s user 0.004s sys 0.020s
I20260927 23:31:46.091822 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.124310 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:46.119570 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.124961 1663 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 9.
I20260927 23:31:46.120512 1655 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:46.029505 4640 consensus_queue.cc:260] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:46.118229 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 50941.5us
I20260927 23:31:46.126541 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:46.129005 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:46.124337 1702 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.092334 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.130709 1730 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:46.129657 1662 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
W20260927 23:31:46.139537 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 48166.7us
I20260927 23:31:46.140429 1700 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.143934 1663 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:46.145577 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.146095 1655 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.091572 4715 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:46.153709 1662 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260927 23:31:46.154078 1662 pending_rounds.cc:76] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 4
I20260927 23:31:46.154273 1662 pending_rounds.cc:97] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260927 23:31:46.148716 1628 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:46.149327 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.149821 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.150693 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:46.110806 4572 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.152155 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.131683 4594 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:46.117476 4209 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.165055 4787 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:46.172593 4787 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:46.153139 1579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1634) took 2795 ms (client timeout 2111 ms). Trace:
I20260927 23:31:46.172933 4787 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
W20260927 23:31:46.173085 1579 rpcz_store.cc:269] 0927 23:31:43.353317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.464619 (+111302us) service_pool.cc:227] Handling call
0927 23:31:46.148628 (+2684009us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540007dd54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:46.148753 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070036352}
I20260927 23:31:46.175055 1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 468) took 2993 ms. Trace:
I20260927 23:31:46.175551 1729 rpcz_store.cc:276] 0927 23:31:43.181950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.600106 (+418156us) service_pool.cc:227] Handling call
0927 23:31:43.665023 (+ 64917us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:44.520258 (+855235us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.174738 (+1654480us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400105754. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.174967 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1902375680}
I20260927 23:31:46.165233 4765 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:46.178813 4765 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.179273 4765 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.180792 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1558) took 2902 ms. Trace:
I20260927 23:31:46.181180 1588 rpcz_store.cc:276] 0927 23:31:43.278009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.695006 (+416997us) service_pool.cc:227] Handling call
0927 23:31:43.834919 (+139913us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:43.935069 (+100150us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.180597 (+2245528us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001fead0. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:46.180715 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242406656}
I20260927 23:31:46.165352 4263 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.182117 4263 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:46.182489 4263 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.184556 4641 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:46.155318 1628 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:46.151010 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.188125 1733 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 9.
I20260927 23:31:46.154502 1662 raft_consensus.cc:2866] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:46.156424 1737 raft_consensus.cc:2396] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:46.156644 1587 raft_consensus.cc:1226] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 23:31:46.159454 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.189359 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1676) took 3206 ms. Trace:
I20260927 23:31:46.160518 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.182528 1663 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:46.167846 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.192916 2257 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.169687 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.189898 1730 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:46.092844 4768 consensus_queue.cc:1225] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer 0fe98f486bab4d0b9c421c145d83f1ce log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260927 23:31:46.159660 4764 raft_consensus.cc:2726] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.171542 4594 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.190248 1586 raft_consensus.cc:1226] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 10. Current term is 11. Ops: [8.8-10.9]
I20260927 23:31:46.146992 4500 raft_consensus.cc:2726] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.195351 1735 raft_consensus.cc:1226] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.246026 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.249861 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.250550 1655 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:46.250710 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.251361 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.252046 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.252772 1737 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:46.253623 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.195573 1628 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.154464 4677 consensus_queue.cc:1046] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.100s
W20260927 23:31:46.254473 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 58978.4us
I20260927 23:31:46.055159 4687 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.255378 1628 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:46.132894 4686 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.150862 4695 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:46.256379 4695 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.256925 4695 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:46.256305 4686 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.259032 4686 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.259923 4811 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:46.260587 4811 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.260952 4811 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.261233 4811 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.131976 4595 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:46.262651 4595 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.265018 4595 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.146509 4807 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:46.265700 4807 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.266175 4807 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:46.266542 1728 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:46.146792 4689 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:46.164129 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.199687 1654 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:46.189901 1658 rpcz_store.cc:276] 0927 23:31:42.954003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:42.954585 (+ 582us) service_pool.cc:227] Handling call
0927 23:31:42.955057 (+ 472us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:43.139566 (+184509us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 23:31:43.648669 (+509103us) log.cc:842] Serialized 14 byte log entry
0927 23:31:44.045565 (+396896us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:44.045612 (+ 47us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:44.045657 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.037286 (+1991629us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:46.159916 (+122630us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001cfcd4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.159985 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:46.160133 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55229,"prepare.queue_time_us":29876,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"raft.queue_time_us":20970,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":10753,"spinlock_wait_cycles":513751168,"thread_start_us":900608,"threads_started":2,"wal-append.queue_time_us":1258297}
I20260927 23:31:46.270364 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.222028 1661 raft_consensus.cc:3032] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:46.091303 4264 consensus_queue.cc:1046] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260927 23:31:46.168433 4715 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.168653 4572 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.230088 4594 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:46.230643 4400 raft_consensus.cc:2726] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:46.233999 2258 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.237195 4814 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.155997 4628 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260927 23:31:46.241612 4818 log.cc:924] Time spent T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20260927 23:31:46.157012 4732 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.242151 4646 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.244288 1733 raft_consensus.cc:1261] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:46.244735 1660 raft_consensus.cc:2396] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260927 23:31:46.283360 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.245146 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.274966 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.284345 1734 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
I20260927 23:31:46.284760 1737 raft_consensus.cc:2354] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:46.285384 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.286046 1734 raft_consensus.cc:1799] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Handling vote request from an unknown peer 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.248495 4674 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.271481 1661 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.268680 1731 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:46.279124 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.278456 4689 raft_consensus.cc:686] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:46.280296 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.247596 4775 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.278209 4264 raft_consensus.cc:3032] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 LEADER]: Stepping down as leader of term 10
I20260927 23:31:46.301985 4722 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260927 23:31:46.336047 4264 raft_consensus.cc:729] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.299937 4646 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.336979 4264 consensus_queue.cc:260] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.337744 4775 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.337863 1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1648) took 2368 ms. Trace:
I20260927 23:31:46.338083 4646 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.276168 4572 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.338936 1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 502) took 2459 ms. Trace:
I20260927 23:31:46.339267 1587 rpcz_store.cc:276] 0927 23:31:43.879496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.600244 (+720748us) service_pool.cc:227] Handling call
0927 23:31:46.150434 (+1550190us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:46.195731 (+ 45297us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.338758 (+143027us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540007dd50. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ea09 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:46.338883 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3564179200}
I20260927 23:31:46.340687 4715 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:46.341630 4716 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.299784 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.343405 1730 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.343546 4716 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:46.278481 4814 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.344779 4814 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.338290 1736 rpcz_store.cc:276] 0927 23:31:43.969144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.548510 (+579366us) service_pool.cc:227] Handling call
0927 23:31:44.558510 (+ 10000us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:45.983642 (+1425132us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.337639 (+353997us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001d1d54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.337806 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3270251392}
I20260927 23:31:46.279305 4628 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.348804 4628 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.278249 4670 raft_consensus.cc:2726] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:46.336078 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.358166 1663 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:46.359453 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.359719 4640 raft_consensus.cc:3037] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:46.359805 1659 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.359826 4557 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:46.362887 4557 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.364010 4674 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.306421 4687 leader_election.cc:288] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.286130 1737 raft_consensus.cc:1261] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:46.338238 4264 raft_consensus.cc:3037] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:46.366377 4557 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.367026 4263 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.367686 4628 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.368867 4263 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.369477 4722 consensus_queue.cc:1046] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:46.370373 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:46.325433 1734 raft_consensus.cc:2354] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:46.326110 2258 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:46.330034 4781 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 1.429s user 0.007s sys 0.023s
W20260927 23:31:46.380581 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 54539.4us
I20260927 23:31:46.275215 4569 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:46.370515 4722 raft_consensus.cc:3032] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:46.372020 1729 raft_consensus.cc:1226] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.303481 4810 raft_consensus.cc:686] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:46.390527 1661 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.382977 1659 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:46.392890 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.340705 4787 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.403628 4787 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.399334 4695 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.405757 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:46.406929 4485 consensus_queue.cc:1046] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:46.409991 1661 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.299997 4732 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.394239 4700 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:46.389730 4569 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:46.393597 4722 raft_consensus.cc:729] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:46.406706 4695 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.412549 4670 raft_consensus.cc:3032] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:46.432883 1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 508) took 2072 ms. Trace:
I20260927 23:31:46.446657 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:46.449651 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 293587us
I20260927 23:31:46.437917 1734 raft_consensus.cc:1261] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260927 23:31:46.428937 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 407573us
I20260927 23:31:46.461048 1585 rpcz_store.cc:276] 0927 23:31:44.361099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.817191 (+456092us) service_pool.cc:227] Handling call
0927 23:31:44.858740 (+ 41549us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.946430 (+ 87690us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:44.946466 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:44.946506 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:44.946546 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.432678 (+1486132us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b5400467654. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.432725 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:46.432831 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":201175936}
I20260927 23:31:46.446686 4687 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:46.423975 4700 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:46.442306 4791 log.cc:924] Time spent T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 1.478s user 0.004s sys 0.024s
I20260927 23:31:46.474478 2293 raft_consensus.cc:1226] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:46.449955 4569 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:46.475349 2286 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48296 (request call id 582) took 1982 ms. Trace:
I20260927 23:31:46.475740 2286 rpcz_store.cc:276] 0927 23:31:44.493458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:44.513636 (+ 20178us) service_pool.cc:227] Handling call
0927 23:31:44.554993 (+ 41357us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:44.611432 (+ 56439us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:44.612602 (+ 1170us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:44.649157 (+ 36555us) log.cc:842] Serialized 26 byte log entry
0927 23:31:44.939579 (+290422us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:44.945007 (+ 5428us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:44.952648 (+ 7641us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:46.441811 (+1489163us) spinlock_profiling.cc:243] Waited 78.7 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:46.475069 (+ 33258us) raft_consensus.cc:1671] finished
0927 23:31:46.475092 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:46.475296 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1534,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"spinlock_wait_cycles":181048192,"thread_start_us":113745,"threads_started":1,"wal-append.queue_time_us":224252}
I20260927 23:31:46.464419 4687 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.418475 4765 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.444018 1583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1545) took 3284 ms. Trace:
I20260927 23:31:46.478892 1583 rpcz_store.cc:276] 0927 23:31:43.159602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:43.217342 (+ 57740us) service_pool.cc:227] Handling call
0927 23:31:43.240603 (+ 23261us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:44.945891 (+1705288us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.443815 (+1497924us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400243454. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:46.443969 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3916333312}
I20260927 23:31:46.479811 4687 leader_election.cc:288] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.479892 4765 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.468554 4704 consensus_queue.cc:1035] T 3565b68068904064a965cd0a82a20802 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260927 23:31:46.481227 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:46.482195 4557 consensus_queue.cc:1046] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:46.483186 4557 raft_consensus.cc:3032] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:46.483481 4557 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.484210 4557 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.485353 4557 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:46.485481 1734 raft_consensus.cc:1226] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.487901 4704 consensus_queue.cc:1046] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:31:46.488752 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.490784 4700 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:46.429768 4814 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.469195 4722 consensus_queue.cc:260] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.419740 4808 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.493469 4808 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:46.493845 4808 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.493906 4722 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
W20260927 23:31:46.497181 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 345106us
I20260927 23:31:46.432813 4811 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.467897 1661 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:46.471803 4810 consensus_queue.cc:237] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.508543 1583 raft_consensus.cc:1226] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.509482 1702 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.515308 4823 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:46.518539 1702 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.520103 4823 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.498991 1551 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.460953 4670 raft_consensus.cc:729] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:46.521853 4598 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:46.440732 4689 consensus_queue.cc:237] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.522218 4598 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.522133 4670 consensus_queue.cc:260] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.523424 4670 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.440557 4628 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:46.442001 4732 leader_election.cc:288] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.501595 2288 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:46.526221 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:46.506032 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 347556us
I20260927 23:31:46.522779 4598 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:46.530591 1554 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:46.531986 1656 raft_consensus.cc:2396] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 8 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260927 23:31:46.533819 4598 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.471184 1700 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.535854 4598 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:46.514365 4637 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260927 23:31:46.551635 2288 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260927 23:31:46.500205 4572 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:46.555830 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 84719.1us
I20260927 23:31:46.557503 4263 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 23:31:46.558481 1700 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.561259 4628 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:46.561342 4765 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.561807 4628 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:46.562574 1734 raft_consensus.cc:1226] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:46.562638 4628 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.565966 4811 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.566293 1552 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.567214 1700 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.552228 4732 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:46.571996 4821 consensus_queue.cc:1046] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:46.572280 1702 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.501333 4807 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.573601 1552 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.581342 1552 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.554111 4705 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:46.587641 1656 raft_consensus.cc:1226] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
W20260927 23:31:46.593220 1657 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.012s sys 0.008s
W20260927 23:31:46.593552 1657 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.012s sys 0.008s
I20260927 23:31:46.590886 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.567629 4572 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:46.597831 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.599802 1654 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.593847 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.600773 1656 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 7: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:46.600595 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.600859 1662 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 9. Current term is 10. Ops: []
I20260927 23:31:46.601176 1655 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:46.601574 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.591868 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:46.587775 4743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.230s user 0.021s sys 0.030s
I20260927 23:31:46.571997 4732 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:46.606019 1663 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:46.573202 4765 raft_consensus.cc:2781] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:46.580178 4389 raft_consensus.cc:2726] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.593729 4611 raft_consensus.cc:3032] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:46.513264 4569 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.616871 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.623715 1659 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:46.624984 4572 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:46.625344 4705 raft_consensus.cc:686] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:46.625440 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 44555.8us
I20260927 23:31:46.566449 4598 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260927 23:31:46.566965 4209 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260927 23:31:46.567674 4733 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260927 23:31:46.626353 1552 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:46.626504 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 78655.5us
W20260927 23:31:46.606834 4743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.249s user 0.021s sys 0.030s
I20260927 23:31:46.625578 4572 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.632212 4598 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:46.633605 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.635519 4807 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:46.623334 1702 connection.cc:232] RPC call timeout handler was delayed by -1.9853402799999458s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20260927 23:31:46.638602 4827 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:46.619611 4732 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:46.619131 4824 raft_consensus.cc:979] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower a2a3317e8da742d9aefe4c251abd4566 to start an election
I20260927 23:31:46.650861 1551 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.657490 1654 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:46.664906 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.629485 4765 raft_consensus.cc:686] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:46.665593 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.666599 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.679032 1732 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:46.618074 4811 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260927 23:31:46.637422 4766 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:46.663779 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.662658 4598 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.671463 4565 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:46.623198 4611 raft_consensus.cc:729] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:46.672041 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:46.666239 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.678391 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.692680 1736 raft_consensus.cc:2379] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:46.693660 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.694250 1735 raft_consensus.cc:2454] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:46.694931 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.679868 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.667232 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.681245 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.698874 1730 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:46.699023 1737 raft_consensus.cc:2379] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:46.699690 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.699469 4765 consensus_queue.cc:237] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.672250 4687 consensus_queue.cc:1035] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260927 23:31:46.669987 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 43740.9us
I20260927 23:31:46.701505 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:46.695976 4565 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.702749 4565 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.706876 4611 consensus_queue.cc:260] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.668632 4389 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:46.690068 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.718294 1734 raft_consensus.cc:2362] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6. Re-sending same reply.
I20260927 23:31:46.719161 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.648003 4775 consensus_queue.cc:1046] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260927 23:31:46.700507 1730 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:46.707381 4572 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:46.682017 1728 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:46.689152 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.726367 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.699729 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.726940 1730 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:46.727386 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.727669 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.728390 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.729614 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.730405 1728 raft_consensus.cc:2396] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:46.691071 4598 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:46.686973 4810 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:46.714859 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.722385 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.710209 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.725224 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.762506 1736 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:46.732007 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b90254fab234e29ba132c165c446490" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.763908 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.765141 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.766069 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.767525 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.738404 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:46.760686 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 51029.7us
I20260927 23:31:46.760277 1735 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 4.
I20260927 23:31:46.768841 1730 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:46.770030 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:46.770887 2293 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:46.708159 4814 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.687299 4726 consensus_queue.cc:1046] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260927 23:31:46.721016 4766 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:46.796775 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.781837 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:46.776965 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:46.691807 4821 consensus_queue.cc:1046] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.080s
I20260927 23:31:46.801476 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:46.827410 1654 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:46.787175 4833 log.cc:924] Time spent T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20260927 23:31:46.786058 4829 log.cc:924] Time spent T 00000000000000000000000000000000 P c2f29821eafd45fbbcc1b22680919836: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260927 23:31:46.774220 4611 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.836409 4766 raft_consensus.cc:1067] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Signalling peer a2a3317e8da742d9aefe4c251abd4566 to start an election
I20260927 23:31:46.838261 1654 raft_consensus.cc:2454] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:46.843173 2257 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:46.723718 4389 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:46.846339 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 124034us
I20260927 23:31:46.849418 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:46.845906 1627 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.852563 1627 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.853807 1627 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:46.855814 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65018.4us
I20260927 23:31:46.851724 1654 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:46.856560 1662 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:46.775925 4733 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260927 23:31:46.772512 4810 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.856412 1736 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:46.929780 1700 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:46.929646 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 75916us
I20260927 23:31:46.928727 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:46.933064 4814 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.936079 4209 consensus_queue.cc:1046] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
W20260927 23:31:46.942246 4834 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20260927 23:31:46.982096 4628 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 23:31:46.934394 4557 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.942907 4807 raft_consensus.cc:3032] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:46.943522 4640 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:46.983151 4807 raft_consensus.cc:729] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.983246 4640 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:46.984164 4807 consensus_queue.cc:260] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.984290 4640 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:46.985394 4640 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:46.985474 4807 raft_consensus.cc:3037] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:46.966495 1627 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.987313 1627 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:46.988063 4814 raft_consensus.cc:2781] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:46.988453 1627 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:46.988459 4814 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:46.989367 4813 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:46.989464 4814 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.989804 4813 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:46.990165 4813 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:46.971108 1552 leader_election.cc:398] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:46.990694 1552 leader_election.cc:401] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260927 23:31:46.724223 4705 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:46.977210 4793 raft_consensus.cc:3032] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:46.933280 4841 raft_consensus.cc:2741] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260927 23:31:46.981489 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 51793.7us
W20260927 23:31:46.970885 4838 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20260927 23:31:46.992329 1627 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.025417 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.026813 1627 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.029769 1627 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.016676 4793 raft_consensus.cc:729] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:47.016772 4557 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:46.856752 4700 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.010551 1552 leader_election.cc:398] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:47.035044 1552 leader_election.cc:401] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260927 23:31:46.928578 4775 raft_consensus.cc:2726] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:46.948535 4845 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
W20260927 23:31:46.978281 4832 log.cc:924] Time spent T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20260927 23:31:47.021695 1700 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:46.989069 4835 log.cc:924] Time spent T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20260927 23:31:47.031905 4811 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:47.034691 4841 raft_consensus.cc:3032] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:47.064580 4841 raft_consensus.cc:729] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:46.936882 4569 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:46.965582 4646 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:47.065783 4646 consensus_queue.cc:1046] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:47.031751 4637 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:47.032162 4209 raft_consensus.cc:3032] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:47.066834 4209 raft_consensus.cc:729] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:47.067538 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 45916.7us
I20260927 23:31:47.067780 4810 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.067679 4209 consensus_queue.cc:260] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.068269 1700 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.069031 4687 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:47.032521 4557 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:47.069494 4687 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.069849 4687 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.070056 4628 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:47.031512 4264 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:46.945803 4565 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.045030 4766 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 23:31:47.071033 4264 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:47.071085 4766 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:47.071261 4732 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.071849 4766 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:47.073349 4732 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.073884 4732 raft_consensus.cc:3037] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.068774 4209 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:47.075614 4565 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.076155 4565 consensus_queue.cc:1046] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:47.048182 4733 raft_consensus.cc:3037] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.018062 4826 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260927 23:31:47.072001 4264 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.039176 4793 consensus_queue.cc:260] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:47.079510 4808 consensus_meta.cc:287] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.572s user 0.009s sys 0.000s
I20260927 23:31:47.065418 4841 consensus_queue.cc:260] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.066615 4646 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:47.070252 4605 consensus_queue.cc:1046] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260927 23:31:47.044387 4828 log.cc:924] Time spent T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260927 23:31:47.083123 4646 raft_consensus.cc:3032] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:47.083654 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 414) took 1144 ms. Trace:
I20260927 23:31:47.084967 1657 rpcz_store.cc:276] 0927 23:31:45.939045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:45.939244 (+ 199us) service_pool.cc:227] Handling call
0927 23:31:45.939582 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:45.939958 (+ 376us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:45.939994 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:45.945809 (+ 5815us) log.cc:842] Serialized 26 byte log entry
0927 23:31:46.593907 (+648098us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:46.593944 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:46.594003 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:47.083468 (+489465us) raft_consensus.cc:1671] finished
0927 23:31:47.083487 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:47.083604 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1877,"raft.queue_time_us":3503,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6304,"thread_start_us":617098,"threads_started":1,"wal-append.queue_time_us":829044}
I20260927 23:31:47.083770 4646 raft_consensus.cc:729] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.079736 4826 raft_consensus.cc:3032] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:47.087417 4826 raft_consensus.cc:729] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.087262 4646 consensus_queue.cc:260] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.081974 4808 raft_consensus.cc:504] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.083021 4598 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.091091 2286 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.091768 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.084554 4700 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.092377 1736 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.093554 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.094542 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.095877 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.097316 1737 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.098492 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.102464 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.100672 4598 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.100476 4808 leader_election.cc:288] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.103691 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.105015 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.090050 4826 consensus_queue.cc:260] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.111421 1732 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 23:31:47.082690 4628 raft_consensus.cc:3032] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:47.112301 4628 raft_consensus.cc:729] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:47.032084 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 350562us
I20260927 23:31:47.113101 4628 consensus_queue.cc:260] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.114256 4628 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.117659 4826 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.122888 1551 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.070088 1700 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:47.156858 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 86822.6us
I20260927 23:31:47.157766 1700 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:47.088443 4646 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.139808 1733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:38820
I20260927 23:31:47.174017 1733 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:47.141425 1736 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:47.174472 1733 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:47.150100 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:47.153255 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 309088us
I20260927 23:31:47.132296 4814 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.159484 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.159621 1700 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:47.179294 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e382c82189e743b4ace885d63e5e9560" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.179728 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.180573 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.191251 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.180784 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.181923 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.182101 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.193387 1735 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.184376 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.194960 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.195430 4598 raft_consensus.cc:2726] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260927 23:31:47.195602 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 72824.6us
I20260927 23:31:47.196015 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.196168 1585 raft_consensus.cc:1226] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.136704 4766 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.197297 1585 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:47.198781 4766 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.172765 4716 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:47.183149 4823 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:47.202087 4823 raft_consensus.cc:686] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:47.202510 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 359398us
I20260927 23:31:47.203183 4823 consensus_queue.cc:237] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.206069 4810 raft_consensus.cc:2781] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:47.206624 4810 raft_consensus.cc:686] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:47.209192 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.209842 1654 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:47.190367 1736 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:47.196337 1551 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.217517 1729 raft_consensus.cc:2379] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:47.219302 1730 raft_consensus.cc:1226] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260927 23:31:47.223981 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 64456.8us
I20260927 23:31:47.230064 1663 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.230950 1657 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
I20260927 23:31:47.194963 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.232846 2257 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:47.233253 1586 raft_consensus.cc:1261] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 23:31:47.233561 1554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.028s sys 0.072s
W20260927 23:31:47.234175 1554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.028s sys 0.072s
I20260927 23:31:47.241294 1551 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.244184 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.245990 1659 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:47.201740 4687 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.210887 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddb51781fec46b194174de5f4406def" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.247277 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4c2a41313d4adcb6e037570a9d5882" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:47.294199 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 50118.7us
W20260927 23:31:47.248329 4827 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.011s sys 0.015s
I20260927 23:31:47.305493 1582 raft_consensus.cc:1226] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260927 23:31:47.299886 1551 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.301061 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.246675 4572 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:47.316030 1582 raft_consensus.cc:1261] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:47.248286 4766 raft_consensus.cc:2781] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:47.324918 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.317369 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.302212 1735 raft_consensus.cc:2454] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:47.297974 1733 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.257447 4598 raft_consensus.cc:2726] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:47.374019 1737 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260927 23:31:47.375447 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 75680.1us
I20260927 23:31:47.381371 1551 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:47.339498 1656 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.069s sys 0.103s
W20260927 23:31:47.382288 1656 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.069s sys 0.103s
I20260927 23:31:47.375979 1585 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:47.377525 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87786104826046e7852edc590e194f5d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.183307 4689 consensus_queue.cc:1046] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
W20260927 23:31:47.303097 4824 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.013s sys 0.052s
W20260927 23:31:47.316115 4827 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.012s sys 0.015s
I20260927 23:31:47.393901 1733 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:47.396095 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.398524 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.400991 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.403074 1728 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:47.417284 4732 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
W20260927 23:31:47.408095 4824 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.013s sys 0.052s
W20260927 23:31:47.418993 4843 log.cc:924] Time spent T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.624s user 0.003s sys 0.000s
I20260927 23:31:47.417294 1734 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:47.421595 1584 raft_consensus.cc:2396] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260927 23:31:47.422895 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.423890 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.323679 4572 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:47.424670 4572 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:47.425945 4820 raft_consensus.cc:3032] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 23:31:47.404099 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.315977 4705 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:47.427461 4705 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.427896 4705 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:47.415426 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.431095 1731 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:47.435118 4837 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
I20260927 23:31:47.436501 4263 raft_consensus.cc:979] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce: : Instructing follower a2a3317e8da742d9aefe4c251abd4566 to start an election
I20260927 23:31:47.436731 1588 raft_consensus.cc:1226] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:47.437865 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.439080 1588 raft_consensus.cc:1226] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.440045 1588 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:47.342495 4766 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.440478 4766 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.442404 1580 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 9
I20260927 23:31:47.448712 1580 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
I20260927 23:31:47.207504 4810 consensus_queue.cc:237] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.426301 4820 raft_consensus.cc:729] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:47.454569 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.467451 1737 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:47.458884 1628 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:47.468089 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 56322.6us
I20260927 23:31:47.324570 4670 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:47.462240 1734 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 10.
I20260927 23:31:47.462577 1552 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.465830 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.471760 1586 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
W20260927 23:31:47.483613 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 453854us
I20260927 23:31:47.484732 1627 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.486200 1627 leader_election.cc:302] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.487617 1627 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.491674 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.493659 1627 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.498220 1627 leader_election.cc:302] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.451509 4572 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.505821 1737 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:47.407864 4847 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:47.488505 4775 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:47.512290 1551 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:47.516085 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 53573.9us
I20260927 23:31:47.516149 1627 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.481736 4807 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:47.529094 4572 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:47.476636 4716 raft_consensus.cc:3032] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:47.476841 4670 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.519310 4811 raft_consensus.cc:2726] T 2b90254fab234e29ba132c165c446490 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:47.529814 4775 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.530033 4670 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:47.519573 4674 consensus_queue.cc:1035] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 23:31:47.489089 4765 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:47.499894 4640 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:47.524005 1552 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.498606 4264 raft_consensus.cc:2726] T 861e779ad2d04547bd9be0bfbc2616c5 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 23:31:47.524506 1627 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.525620 1731 raft_consensus.cc:2454] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:47.480180 4598 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260927 23:31:47.510517 4637 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:47.528844 4389 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:47.522146 1553 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.008s sys 0.012s
W20260927 23:31:47.572683 1553 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.009s sys 0.012s
I20260927 23:31:47.499554 4557 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.539844 1586 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9.
W20260927 23:31:47.544260 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 311165us
I20260927 23:31:47.512705 4853 raft_consensus.cc:1067] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Signalling peer a2a3317e8da742d9aefe4c251abd4566 to start an election
I20260927 23:31:47.559319 1552 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.567476 1627 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.523277 4847 raft_consensus.cc:482] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:47.529846 4716 raft_consensus.cc:729] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:47.572268 4775 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 23:31:47.561060 4813 consensus_meta.cc:287] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.551s user 0.006s sys 0.003s
I20260927 23:31:47.551486 4824 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260927 23:31:47.534578 4733 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731) with higher term. Message: Invalid argument: T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:47.591674 1627 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:47.580058 396 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 23:31:47.615010 1627 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.564550 4640 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:47.572530 4811 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:47.576730 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 64547us
I20260927 23:31:47.617429 1627 leader_election.cc:302] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.618891 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.572870 4637 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.616163 2258 leader_election.cc:302] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:47.609599 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50364.4us
I20260927 23:31:47.587221 4263 raft_consensus.cc:2781] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:47.637876 1580 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260927 23:31:47.606508 4557 raft_consensus.cc:2781] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:47.648478 4557 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:47.648881 4557 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:47.606298 4847 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.647327 4844 raft_consensus.cc:2726] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.662048 1552 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.610690 4840 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:47.633210 4807 raft_consensus.cc:2726] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:47.667553 1658 raft_consensus.cc:1261] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:47.598312 4814 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:47.613375 4611 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.761s user 0.010s sys 0.000s
I20260927 23:31:47.652895 1627 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:47.654682 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.655982 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.545894 4820 consensus_queue.cc:260] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.657430 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.658146 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.576601 4766 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.655336 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.654058 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:47.632797 4855 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:47.619653 4687 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:47.637207 4640 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
W20260927 23:31:47.664151 1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35356 (request call id 150) took 11380 ms (client timeout 15000 ms). Trace:
I20260927 23:31:47.671420 1583 raft_consensus.cc:2454] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:47.621420 4813 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.636547 4637 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.666791 4716 consensus_queue.cc:260] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.613768 4827 consensus_queue.cc:1046] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260927 23:31:47.640023 4775 raft_consensus.cc:2726] T b9fe23b5411b441fbd8792ef9dc4e99b P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260927 23:31:47.672502 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 56431.9us
I20260927 23:31:47.714474 4807 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:47.740552 4807 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.741712 1584 raft_consensus.cc:2379] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:47.696100 4565 consensus_queue.cc:1046] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260927 23:31:47.746009 4854 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.666190 4853 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260927 23:31:47.749635 4775 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:47.750162 4775 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.751943 4793 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.752379 4775 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
W20260927 23:31:47.756565 4850 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20260927 23:31:47.668565 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.757756 1656 raft_consensus.cc:2379] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 10: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:47.757994 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.759160 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:47.697870 1662 raft_consensus.cc:2379] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 9: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
W20260927 23:31:47.702479 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 49659.4us
I20260927 23:31:47.641191 4598 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260927 23:31:47.711424 1659 raft_consensus.cc:2454] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:47.759720 2288 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:47.764974 1659 raft_consensus.cc:1226] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260927 23:31:47.765202 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 99014.9us
I20260927 23:31:47.767153 1703 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:47.768374 1656 raft_consensus.cc:1226] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260927 23:31:47.712646 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.639664 4811 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.709416 4674 consensus_queue.cc:1035] T e5355e82529343f5b479b2450cefa8ea P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:47.719691 4264 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:47.710640 4700 raft_consensus.cc:2781] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:47.724215 4705 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.730759 4716 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.732735 4813 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:47.734939 4858 log.cc:924] Time spent T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260927 23:31:47.736979 4847 raft_consensus.cc:504] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.679579 4851 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:47.739085 4766 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.702415 1484 catalog_manager.cc:5495] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 5, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:47.665105 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.761411 1662 raft_consensus.cc:1261] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260927 23:31:47.792663 1662 pending_rounds.cc:76] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 8
I20260927 23:31:47.792943 1662 pending_rounds.cc:97] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260927 23:31:47.793190 1662 raft_consensus.cc:2866] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 23:31:47.702972 4857 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260927 23:31:47.763932 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:47.777693 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:47.794382 1581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.031s sys 0.037s
I20260927 23:31:47.699712 4814 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.792797 1484 catalog_manager.cc:5495] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 4 to 5, leader changed from 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:47.796082 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 397649us
I20260927 23:31:47.796268 1655 raft_consensus.cc:2379] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:47.796711 2286 raft_consensus.cc:2379] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 5: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:47.796512 1484 catalog_manager.cc:5495] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 7, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 7 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:47.668442 4263 raft_consensus.cc:686] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:47.705752 1486 rpcz_store.cc:269] 0927 23:31:36.278692 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:36.278898 (+ 206us) service_pool.cc:227] Handling call
0927 23:31:47.659033 (+11380135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 23:31:40.735819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 23:31:40.891801 (+155982us) write_op.cc:432] Acquiring schema lock in shared mode
0927 23:31:40.898846 (+ 7045us) write_op.cc:435] Acquired schema lock
0927 23:31:40.898877 (+ 31us) tablet.cc:660] Decoding operations
0927 23:31:41.667209 (+768332us) write_op.cc:620] Acquiring the partition lock for write op
0927 23:31:41.705862 (+ 38653us) write_op.cc:641] Partition lock acquired for write op
0927 23:31:41.705908 (+ 46us) tablet.cc:683] Acquiring locks for 19 operations
0927 23:31:41.707871 (+ 1963us) tablet.cc:699] Row locks acquired
0927 23:31:41.707914 (+ 43us) write_op.cc:260] PREPARE: finished
0927 23:31:41.708138 (+ 224us) write_op.cc:270] Start()
0927 23:31:41.708327 (+ 189us) write_op.cc:276] Timestamp: P: 1790551901708053 usec, L: 0
0927 23:31:41.708352 (+ 25us) op_driver.cc:331] REPLICATION: starting
0927 23:31:41.709196 (+ 844us) log.cc:842] Serialized 7316 byte log entry
0927 23:31:43.474460 (+1765264us) op_driver.cc:444] REPLICATION: finished
0927 23:31:44.005014 (+530554us) write_op.cc:301] APPLY: starting
0927 23:31:44.009932 (+ 4918us) tablet.cc:1365] starting BulkCheckPresence
0927 23:31:44.106386 (+ 96454us) tablet.cc:1368] finished BulkCheckPresence
0927 23:31:44.183206 (+ 76820us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 23:31:44.357022 (+173816us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 23:31:44.357343 (+ 321us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
0927 23:31:44.357390 (+ 47us) write_op.cc:312] APPLY: finished
0927 23:31:44.357756 (+ 366us) log.cc:842] Serialized 132 byte log entry
0927 23:31:46.060883 (+1703127us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f974db295fe 00007f9756960a39 00007f974db468bb 00007f974db44cbf 00007f974db46e22 00007f97506f05bb
0927 23:31:46.669480 (+608597us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:46.684273 (+ 14793us) write_op.cc:454] Released schema lock
0927 23:31:46.690033 (+ 5760us) write_op.cc:341] FINISH: Updating metrics
0927 23:31:46.748665 (+ 58632us) write_op.cc:489] Releasing partition, row and schema locks
0927 23:31:46.767705 (+ 19040us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":395434,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":431121,"apply.run_cpu_time_us":58399,"apply.run_wall_time_us":2803409,"num_ops":19,"prepare.queue_time_us":160681,"prepare.run_cpu_time_us":19413,"prepare.run_wall_time_us":1281720,"raft.queue_time_us":796296,"raft.run_cpu_time_us":14515,"raft.run_wall_time_us":327564,"replication_time_us":1758244,"spinlock_wait_cycles":14824704,"thread_start_us":3325912,"threads_started":5,"wal-append.queue_time_us":2957760,"wal-append.run_cpu_time_us":46190,"wal-append.run_wall_time_us":1833779}]]}
W20260927 23:31:47.794734 1581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.032s sys 0.038s
I20260927 23:31:47.794729 1657 raft_consensus.cc:3037] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:47.797907 1552 leader_election.cc:302] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:47.801729 1660 raft_consensus.cc:1799] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Handling vote request from an unknown peer 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.801810 1702 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.802448 1660 raft_consensus.cc:2354] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:47.797847 1484 catalog_manager.cc:5495] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: config changed from index 3 to 6, term changed from 3 to 7, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to <none>, VOTER 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) evicted. New cstate: current_term: 7 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:47.805229 1582 raft_consensus.cc:1226] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
I20260927 23:31:47.811421 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.813196 1656 raft_consensus.cc:1261] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:47.774266 4689 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:47.814154 1552 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:47.815539 1655 raft_consensus.cc:1261] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 23:31:47.815948 4811 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.816066 4841 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.816658 4839 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:47.845880 4637 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.846486 4687 consensus_queue.cc:1046] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:47.799558 4813 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:47.847429 4813 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.848395 4637 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.849061 4839 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:47.849517 4839 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:47.849805 4813 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:47.817006 1585 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:47.805204 4766 consensus_queue.cc:1046] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260927 23:31:47.814433 4861 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:47.851754 4766 raft_consensus.cc:3032] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:47.852200 4766 raft_consensus.cc:729] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.853085 4766 consensus_queue.cc:260] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.856438 4670 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.780730 4264 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.861306 4557 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.862947 4263 consensus_queue.cc:237] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.813526 4851 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.868767 1587 raft_consensus.cc:2396] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:47.868317 4851 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.828900 4689 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:47.871210 4689 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.820942 1657 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 10.
I20260927 23:31:47.822818 4733 raft_consensus.cc:2781] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Leader election won for term 10
I20260927 23:31:47.826100 1484 catalog_manager.cc:5495] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 7, leader changed from 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 7 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 23:31:47.828099 1702 leader_election.cc:302] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.832238 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.833084 1663 raft_consensus.cc:1261] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 23:31:47.876803 1662 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.833791 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:47.878084 1484 catalog_manager.cc:5495] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:47.839085 1656 raft_consensus.cc:1226] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:47.795435 4700 raft_consensus.cc:686] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:47.778594 4807 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:47.840685 4674 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader election won for term 4
I20260927 23:31:47.840878 4847 leader_election.cc:288] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.841807 4646 raft_consensus.cc:3032] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
W20260927 23:31:47.898977 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 71802.8us
I20260927 23:31:47.900019 1702 leader_election.cc:302] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:47.902017 1702 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.885301 4766 raft_consensus.cc:3037] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.880931 1484 catalog_manager.cc:5495] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:47.804765 4732 consensus_queue.cc:1046] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260927 23:31:47.888196 1659 raft_consensus.cc:1226] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 4. Current term is 5. Ops: []
I20260927 23:31:47.889091 4854 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:47.888866 4700 consensus_queue.cc:237] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:47.972013 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 70065.3us
I20260927 23:31:47.972645 4628 raft_consensus.cc:2726] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 23:31:47.891070 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 49294.1us
I20260927 23:31:47.973268 1702 leader_election.cc:302] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:47.927377 1628 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.99.1:45501 timed out after 1.517s (SENT) [suppressed 33 similar messages]
I20260927 23:31:47.899643 1662 raft_consensus.cc:2454] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:47.888944 4820 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:47.934434 1484 catalog_manager.cc:5495] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 5 to 7, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 7 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:47.834962 1552 leader_election.cc:302] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:47.967690 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.964284 4572 raft_consensus.cc:2726] T 6e4c2a41313d4adcb6e037570a9d5882 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:47.964454 4860 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:47.983398 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 148532us
I20260927 23:31:47.968449 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.984166 1729 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:47.969089 1731 raft_consensus.cc:1226] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:47.968936 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:47.969638 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.969924 1728 raft_consensus.cc:1226] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 3. Current term is 4. Ops: []
I20260927 23:31:47.970330 4716 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:47.893430 4861 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:47.967127 1660 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:47.895998 4687 raft_consensus.cc:3032] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:47.996981 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:47.971728 4854 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.002425 4854 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.003603 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.004738 1582 raft_consensus.cc:2379] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:48.005368 4444 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.006129 1583 raft_consensus.cc:1226] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
I20260927 23:31:48.005923 4444 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:47.997856 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.007928 1702 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:47.967051 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.977378 1484 catalog_manager.cc:5495] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 4 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.009712 1484 catalog_manager.cc:5495] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 4 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:47.895727 4646 raft_consensus.cc:729] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.978580 1735 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:47.891458 4264 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:47.892292 4733 raft_consensus.cc:686] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:47.970620 1732 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
W20260927 23:31:47.966223 4826 consensus_meta.cc:287] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.746s user 0.001s sys 0.010s
I20260927 23:31:48.029476 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.982354 4605 raft_consensus.cc:2726] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:47.931097 4823 raft_consensus.cc:2726] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:47.964147 4674 raft_consensus.cc:686] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.033325 1579 raft_consensus.cc:2396] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260927 23:31:47.931672 4807 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.021608 1587 raft_consensus.cc:3032] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:48.016906 1733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b"
dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:38820
I20260927 23:31:48.029618 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:48.033975 1587 raft_consensus.cc:729] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.011758 1484 catalog_manager.cc:5495] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.035661 1484 catalog_manager.cc:5495] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 3, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4). New cstate: current_term: 3 leader_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:47.995890 4827 raft_consensus.cc:2781] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:47.992919 4637 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.007966 4687 raft_consensus.cc:729] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.037799 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:47.973305 4628 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260927 23:31:48.038772 1484 catalog_manager.cc:5495] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 5 to 6, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 6 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.035715 1587 consensus_queue.cc:260] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:47.980082 4389 consensus_queue.cc:1046] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260927 23:31:48.045244 1587 raft_consensus.cc:3037] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:47.997151 4765 consensus_queue.cc:1046] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260927 23:31:48.051491 1484 catalog_manager.cc:5495] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 4 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.058399 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:48.060369 1728 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:47.979384 4862 consensus_queue.cc:1035] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260927 23:31:48.068182 1484 catalog_manager.cc:5495] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.012779 4572 consensus_queue.cc:1046] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260927 23:31:48.075870 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:48.077841 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:48.013525 4861 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.084293 4861 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:48.064572 4646 consensus_queue.cc:260] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.052713 4827 raft_consensus.cc:686] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.092376 4827 consensus_queue.cc:237] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.094455 4209 consensus_queue.cc:1046] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:48.074862 4823 consensus_queue.cc:1035] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260927 23:31:48.039947 4674 consensus_queue.cc:237] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.098824 4628 raft_consensus.cc:3032] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:48.099326 4628 raft_consensus.cc:729] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.100232 4628 consensus_queue.cc:260] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.011880 4716 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.072475 1484 catalog_manager.cc:5495] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.064317 4670 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.102023 1733 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:48.106979 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.105432 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:48.107546 1733 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:48.106151 1484 catalog_manager.cc:5495] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 6, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 6 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:48.043732 4637 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.108991 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:47.997437 4860 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.119815 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.120723 1579 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:48.120299 1484 catalog_manager.cc:5495] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 4 to 5, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3). New cstate: current_term: 5 leader_uuid: "a2a3317e8da742d9aefe4c251abd4566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } health_report { overall_health: HEALTHY } } }
I20260927 23:31:48.124471 1484 catalog_manager.cc:5495] T e5355e82529343f5b479b2450cefa8ea P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 3, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.040053 4807 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.070549 1587 raft_consensus.cc:2454] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 8.
I20260927 23:31:48.103312 4765 raft_consensus.cc:2726] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:48.126916 1484 catalog_manager.cc:5495] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 4, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
I20260927 23:31:48.142990 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.128782 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.147099 1583 raft_consensus.cc:2379] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 9: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:48.146154 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:48.146283 1584 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:48.150935 1484 catalog_manager.cc:5495] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 7, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 7 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.154867 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.155786 1732 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:48.104406 4705 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.108563 4389 raft_consensus.cc:3032] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:48.162176 1733 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.165753 1735 raft_consensus.cc:2454] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:48.168109 1728 raft_consensus.cc:2454] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 10.
I20260927 23:31:48.170477 1484 catalog_manager.cc:5495] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 3 to 4, leader changed from 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.111454 4444 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.177209 1729 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260927 23:31:48.121739 4861 raft_consensus.cc:3032] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:48.177583 2258 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:48.181628 2258 leader_election.cc:302] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:48.111317 4628 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
W20260927 23:31:48.159693 4854 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260927 23:31:48.138147 4557 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:48.181098 1654 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.011s sys 0.030s
W20260927 23:31:48.189549 1654 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s user 0.011s sys 0.030s
W20260927 23:31:48.185087 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.006s sys 0.035s
W20260927 23:31:48.191375 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s user 0.007s sys 0.035s
W20260927 23:31:48.191748 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 573542us
W20260927 23:31:48.187772 2257 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.013s sys 0.029s
W20260927 23:31:48.195570 2257 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.556s user 0.013s sys 0.029s
I20260927 23:31:48.179271 1484 catalog_manager.cc:5495] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.171120 4389 raft_consensus.cc:729] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.212616 1551 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:48.185256 4861 raft_consensus.cc:729] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.124199 4687 consensus_queue.cc:260] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.214087 1551 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.220372 1551 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.216662 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.191956 4722 raft_consensus.cc:2781] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 23:31:48.217262 2293 raft_consensus.cc:1226] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:48.221338 2288 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:48.192330 4726 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:48.230500 2294 raft_consensus.cc:1261] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 23:31:48.249919 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 251265us
W20260927 23:31:48.235124 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 307770us
I20260927 23:31:48.211042 1484 catalog_manager.cc:5495] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 reported cstate change: term changed from 1 to 5, leader changed from a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) to 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2). New cstate: current_term: 5 leader_uuid: "1b945905616b4989853b0a92bfbd6b98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } }
I20260927 23:31:48.232470 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.222106 4598 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:48.133610 4733 consensus_queue.cc:237] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.251905 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bbde2432f547cbab3588c16fb623cf" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.269688 1700 leader_election.cc:302] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:48.256803 2294 raft_consensus.cc:1226] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 3. Current term is 4. Ops: []
W20260927 23:31:48.257675 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 249818us
I20260927 23:31:48.261900 4765 consensus_queue.cc:1035] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:48.263314 2292 raft_consensus.cc:2379] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:48.226864 4847 raft_consensus.cc:2726] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:48.273818 1700 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:48.233400 4726 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 23:31:48.275022 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 65647.8us
I20260927 23:31:48.264873 1628 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:48.236735 4855 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.009s sys 0.009s
I20260927 23:31:48.232553 4722 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:48.264428 4264 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:48.285856 4868 log.cc:924] Time spent T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260927 23:31:48.284020 2259 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.266337 4826 raft_consensus.cc:2726] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:48.288568 4389 consensus_queue.cc:260] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.269132 4637 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.268548 4557 raft_consensus.cc:482] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:48.281599 4572 raft_consensus.cc:3032] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 23:31:48.284410 4726 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:48.297626 4716 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:48.296800 4855 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.009s sys 0.010s
I20260927 23:31:48.315735 4572 raft_consensus.cc:729] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.321700 4864 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.011s
I20260927 23:31:48.340029 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.340840 1729 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:48.340328 4572 consensus_queue.cc:260] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.332235 4389 raft_consensus.cc:3037] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:48.341459 4572 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:48.342350 4716 consensus_queue.cc:1046] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:48.343344 4765 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:48.343806 4765 raft_consensus.cc:686] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.344586 4765 consensus_queue.cc:237] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:48.346474 4872 log.cc:924] Time spent T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260927 23:31:48.346488 4722 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.348842 4572 raft_consensus.cc:3032] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:48.350170 4572 raft_consensus.cc:729] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.352241 4572 consensus_queue.cc:260] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.283756 4870 consensus_queue.cc:1035] T e850f344b805403cb981ec2cf5272ac2 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260927 23:31:48.316125 4670 raft_consensus.cc:2726] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260927 23:31:48.325284 4640 consensus_meta.cc:287] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98: Time spent flushing consensus metadata: real 0.609s user 0.010s sys 0.000s
W20260927 23:31:48.348814 4875 log.cc:924] Time spent T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260927 23:31:48.325517 4802 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:48.367666 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 90344.5us
I20260927 23:31:48.284447 4861 consensus_queue.cc:260] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.322904 4860 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:31:48.304057 4264 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.389814 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:48.330988 4807 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.394172 1702 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:48.319633 4557 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.338966 4855 raft_consensus.cc:686] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:48.366482 4883 raft_consensus.cc:2781] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:48.413259 4884 raft_consensus.cc:2726] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:48.394527 4802 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.409158 4893 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:48.418090 4822 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.422961 4883 raft_consensus.cc:686] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.410982 4765 consensus_queue.cc:1046] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260927 23:31:48.442094 4802 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:48.441460 4839 consensus_meta.cc:287] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.592s user 0.008s sys 0.004s
W20260927 23:31:48.487255 1538 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.003s sys 0.025s
W20260927 23:31:48.488317 1538 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.629s user 0.004s sys 0.025s
I20260927 23:31:48.508832 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.510627 1579 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:48.519930 1585 raft_consensus.cc:1226] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 LEADER]: Rejecting Update request from peer 1b945905616b4989853b0a92bfbd6b98 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:48.522423 1586 raft_consensus.cc:2396] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260927 23:31:48.523321 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.526175 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
W20260927 23:31:48.529034 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 264175us
I20260927 23:31:48.538677 1628 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:48.543440 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 269654us
I20260927 23:31:48.543835 1628 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.546053 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:48.546510 1702 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:48.387490 4640 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.570549 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.571691 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.573071 2294 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:48.578832 4637 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:48.574059 4640 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:48.595968 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 50447.6us
W20260927 23:31:48.602041 4689 consensus_meta.cc:287] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.730s user 0.014s sys 0.001s
I20260927 23:31:48.602746 4689 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.580107 4637 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260927 23:31:48.606890 4802 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.607779 4802 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.609047 4674 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.608970 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.610127 1584 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:48.609920 4674 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.610828 4877 consensus_queue.cc:1046] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 23:31:48.612581 4674 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.613054 4557 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.613999 1734 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:48.614817 4637 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:48.615468 4637 raft_consensus.cc:482] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:48.615912 4637 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:48.615658 4839 raft_consensus.cc:504] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.614741 4879 raft_consensus.cc:482] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.620282 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.621421 1584 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:48.622829 4839 leader_election.cc:288] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.624236 2293 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51538 (request call id 555) took 1417 ms. Trace:
I20260927 23:31:48.624377 4877 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:48.625384 4766 consensus_queue.cc:1046] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:48.644773 4557 raft_consensus.cc:3032] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:48.649699 4557 raft_consensus.cc:729] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:48.651412 4557 consensus_queue.cc:260] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.653749 4557 raft_consensus.cc:3037] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:48.658704 4807 consensus_queue.cc:1046] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:48.660039 1661 raft_consensus.cc:1226] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 7. Current term is 8. Ops: []
I20260927 23:31:48.667714 1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46188 (request call id 1956) took 1012 ms. Trace:
I20260927 23:31:48.668287 1661 rpcz_store.cc:276] 0927 23:31:47.654827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:47.655710 (+ 883us) service_pool.cc:227] Handling call
0927 23:31:47.668909 (+ 13199us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:48.660705 (+991796us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:48.661646 (+ 941us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b5400041ad4. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:48.667545 (+ 5899us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2086706432}
I20260927 23:31:48.627907 2293 rpcz_store.cc:276] 0927 23:31:47.206861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:47.210151 (+ 3290us) service_pool.cc:227] Handling call
0927 23:31:47.281763 (+ 71612us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:48.233119 (+951356us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:48.623768 (+390649us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b5400541354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:48.624043 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2106923264}
I20260927 23:31:48.642701 4689 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.679831 2259 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:48.683048 4806 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:48.662020 4687 raft_consensus.cc:3037] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:48.687968 4826 consensus_queue.cc:1035] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260927 23:31:48.689143 4851 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.678583 4802 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:48.695569 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 151998us
I20260927 23:31:48.695590 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:48.696246 4884 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.701538 2293 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:48.702711 4264 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.701592 4841 consensus_queue.cc:1046] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260927 23:31:48.705333 4895 raft_consensus.cc:2781] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:48.699949 4640 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.715860 4906 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.720582 2293 raft_consensus.cc:2454] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:48.724206 4732 raft_consensus.cc:3032] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:48.724659 4732 raft_consensus.cc:729] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.724665 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.724972 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.697086 4877 raft_consensus.cc:686] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
W20260927 23:31:48.728358 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 100038us
I20260927 23:31:48.733369 4851 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.734742 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:48.738597 1728 raft_consensus.cc:1226] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:48.738843 4841 raft_consensus.cc:3032] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Stepping down as leader of term 3
I20260927 23:31:48.739302 4841 raft_consensus.cc:729] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.739836 4903 consensus_queue.cc:1046] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 23:31:48.740406 4841 consensus_queue.cc:260] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.741335 4903 raft_consensus.cc:3032] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:48.741766 4903 raft_consensus.cc:729] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:48.742300 4841 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Advancing to term 4
I20260927 23:31:48.742578 4851 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.618928 4879 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.440696 4822 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.444547 4883 consensus_queue.cc:237] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.744526 2259 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:48.742527 1628 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.755677 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.756510 4839 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.757344 4839 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.759490 4822 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:48.760010 4839 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.759508 4903 consensus_queue.cc:260] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.762602 4879 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:48.445650 4855 consensus_queue.cc:237] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.763245 4903 raft_consensus.cc:3037] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:48.764735 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:48.765477 2293 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:48.766417 4640 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.766896 1554 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:48.772336 4640 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.732722 4264 raft_consensus.cc:482] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.774806 4264 raft_consensus.cc:504] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.742527 4896 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.725862 4732 consensus_queue.cc:260] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.716576 4906 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.782773 4853 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.783448 4853 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.784121 4906 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:48.776015 4802 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260927 23:31:48.780854 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 560354us
I20260927 23:31:48.786343 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:48.785564 4853 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:48.777715 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 45047us
I20260927 23:31:48.777750 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:48.830286 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 48249.1us
I20260927 23:31:48.780627 4732 raft_consensus.cc:3037] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:48.775738 4700 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:48.843636 1729 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260927 23:31:48.832804 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 49297.8us
I20260927 23:31:48.794811 4802 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:48.778718 4904 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:48.776952 4264 leader_election.cc:288] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.853714 1628 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.856839 4807 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.859020 4807 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.861457 4807 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:48.847373 4700 raft_consensus.cc:482] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:48.862679 4700 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:48.863128 2288 raft_consensus.cc:1226] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 23:31:48.850345 4802 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:48.833741 4640 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:48.872018 1627 leader_election.cc:302] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:48.853415 4904 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.887833 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:48.890560 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.870232 4895 raft_consensus.cc:686] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:48.909262 4637 raft_consensus.cc:504] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.910593 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:48.911839 4847 consensus_queue.cc:1046] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260927 23:31:48.873024 4264 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.776130 4884 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:48.917976 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 65782us
I20260927 23:31:48.779970 4896 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.892992 4904 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:48.929627 4765 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:48.930364 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.931105 4557 raft_consensus.cc:2781] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:48.931923 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.932363 1662 raft_consensus.cc:1226] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 9. Current term is 10. Ops: []
I20260927 23:31:48.932859 1732 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:48.934180 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:48.934474 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 95574.4us
I20260927 23:31:48.935376 1628 leader_election.cc:302] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.870040 4879 consensus_queue.cc:1046] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260927 23:31:48.936761 1661 raft_consensus.cc:1261] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260927 23:31:48.937216 1661 pending_rounds.cc:76] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 8
I20260927 23:31:48.937493 1661 pending_rounds.cc:97] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260927 23:31:48.937772 1661 raft_consensus.cc:2866] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:48.941905 4884 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:48.942282 1733 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:48.942533 4565 raft_consensus.cc:3032] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:48.942991 4565 raft_consensus.cc:729] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:48.944082 4565 consensus_queue.cc:260] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.930231 4765 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.776965 4877 consensus_queue.cc:237] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.960813 4557 raft_consensus.cc:686] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:48.967994 4557 consensus_queue.cc:237] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.968586 1552 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:48.970093 4907 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:48.970817 4907 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:48.989866 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.975775 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:48.945482 4565 raft_consensus.cc:3037] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:48.924190 4640 raft_consensus.cc:2726] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:49.014590 1735 raft_consensus.cc:2379] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:48.984045 4716 raft_consensus.cc:3032] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:48.977284 4839 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.997995 4841 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:48.926146 4847 consensus_queue.cc:1046] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260927 23:31:49.034489 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.035219 1735 raft_consensus.cc:1261] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 23:31:49.037133 4726 consensus_meta.cc:287] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.710s user 0.012s sys 0.003s
I20260927 23:31:49.038030 4807 raft_consensus.cc:2726] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 23:31:48.986236 4862 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:48.974288 4907 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.040164 1731 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:48.965044 4264 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:48.974404 4853 consensus_queue.cc:1035] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260927 23:31:49.064424 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58c5d7cee3a4fa7b7b7d55bf8cd213b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.066391 4765 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.022884 4716 raft_consensus.cc:729] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:49.030391 4881 log.cc:924] Time spent T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.452s user 0.001s sys 0.000s
I20260927 23:31:49.069792 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.074538 1736 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:49.092410 1551 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.113499 2293 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:51840
I20260927 23:31:49.114732 2293 raft_consensus.cc:595] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:49.115319 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.072780 4839 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.057932 4726 raft_consensus.cc:504] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.131524 2288 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:49.132301 4862 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.133517 4700 raft_consensus.cc:504] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.134390 4855 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.134768 4862 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.134948 4855 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.135520 4862 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.127308 4807 raft_consensus.cc:482] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:49.136603 4896 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.137007 4855 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.141156 2286 consensus_queue.cc:237] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:49.141994 4628 consensus_meta.cc:287] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.953s user 0.012s sys 0.009s
I20260927 23:31:49.142146 1581 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:49.147969 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.172551 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.128918 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.124240 4557 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:49.158891 4733 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.159047 4823 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.159734 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.160458 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:49.162891 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 92282.7us
I20260927 23:31:49.163694 4930 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:49.151813 4907 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.165534 4839 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.166389 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.167292 1663 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:49.167423 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.167939 1580 raft_consensus.cc:1261] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260927 23:31:49.162124 4826 raft_consensus.cc:2726] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:49.169593 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.171088 1658 raft_consensus.cc:3032] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 23:31:49.171473 4726 leader_election.cc:288] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.116920 4637 leader_election.cc:288] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.167954 1629 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:49.172778 1579 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.173511 1662 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:49.121276 4895 consensus_queue.cc:237] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.188169 1656 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 10.
I20260927 23:31:49.188086 1658 raft_consensus.cc:729] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:49.180025 4806 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:49.187705 1580 pending_rounds.cc:76] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 3
I20260927 23:31:49.200215 1580 pending_rounds.cc:97] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260927 23:31:49.200517 1580 raft_consensus.cc:2866] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:49.197592 1583 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:49.197041 1657 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:49.197183 4930 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.191745 4733 raft_consensus.cc:504] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:49.193764 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 258455us
I20260927 23:31:49.198235 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.202839 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:49.180621 4557 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.199816 1730 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260927 23:31:49.209132 1730 pending_rounds.cc:76] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 3
I20260927 23:31:49.209482 1730 pending_rounds.cc:97] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260927 23:31:49.205842 1736 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.203478 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:49.210654 1731 raft_consensus.cc:2396] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:49.207804 1587 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 9.
I20260927 23:31:49.142817 4716 consensus_queue.cc:260] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.209725 1730 raft_consensus.cc:2866] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 23:31:49.213615 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 45976.3us
I20260927 23:31:49.211555 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.202072 4806 raft_consensus.cc:3037] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.180473 4862 raft_consensus.cc:504] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.215462 1733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.218768 1581 raft_consensus.cc:3032] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:49.219275 4793 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260927 23:31:49.221140 4793 raft_consensus.cc:482] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.221557 4793 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:49.222540 1551 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.223671 4826 raft_consensus.cc:2781] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 23:31:49.224058 4826 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:49.224406 4826 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:49.213661 4823 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.225044 4640 raft_consensus.cc:3032] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:49.225487 4640 raft_consensus.cc:729] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:49.212212 4557 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.192159 4802 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.220448 1581 raft_consensus.cc:729] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:49.226467 4640 consensus_queue.cc:260] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.234730 1583 raft_consensus.cc:1226] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:49.235244 4722 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.211266 1658 consensus_queue.cc:260] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.226500 4823 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.235800 4722 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.236589 1658 raft_consensus.cc:3037] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:49.239104 1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1837) took 1130 ms. Trace:
I20260927 23:31:49.218405 4637 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:49.192582 4904 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.226622 4716 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.204972 4807 raft_consensus.cc:504] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.239496 1734 rpcz_store.cc:276] 0927 23:31:48.108311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:48.108431 (+ 120us) service_pool.cc:227] Handling call
0927 23:31:48.108777 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:48.614582 (+505805us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:49.238717 (+624135us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54002bcd54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:49.239013 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1160893312}
I20260927 23:31:49.205569 4907 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.253254 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.253508 4572 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:49.238121 4722 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.255275 1583 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 9
I20260927 23:31:49.242231 4637 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.244710 4264 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.261478 1582 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:49.263430 1661 raft_consensus.cc:1226] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260927 23:31:49.258818 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.259814 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.273748 1730 raft_consensus.cc:2396] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 23:31:49.265375 4637 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.265592 4861 raft_consensus.cc:3037] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.287024 4905 raft_consensus.cc:3032] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:49.287562 4905 raft_consensus.cc:729] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:49.288650 4905 consensus_queue.cc:260] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.290609 1655 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:49.291656 4907 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.298686 4700 leader_election.cc:288] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.301157 4905 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.301812 1585 raft_consensus.cc:2396] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260927 23:31:49.302057 4807 leader_election.cc:288] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.302657 4807 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:49.303246 4807 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.305124 4807 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.305470 4637 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.307545 4802 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.308866 4572 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.310181 1731 raft_consensus.cc:1261] T 2bac3d0a2c8444f79b96925750c29b42 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:49.311247 4733 leader_election.cc:288] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.254390 4883 raft_consensus.cc:482] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.278047 1658 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 23:31:49.255218 4931 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:49.279423 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.317634 1736 raft_consensus.cc:2454] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:49.279851 1579 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:49.264707 4640 raft_consensus.cc:3037] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.278693 4823 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.283007 1587 raft_consensus.cc:2379] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:49.275230 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.258560 1581 consensus_queue.cc:260] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.335270 1662 raft_consensus.cc:2379] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 6: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:49.335701 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.336347 1579 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.336395 4862 leader_election.cc:288] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.337102 4557 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.338860 1581 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.340674 1662 raft_consensus.cc:1226] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260927 23:31:49.342054 4883 raft_consensus.cc:504] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.342589 4907 raft_consensus.cc:482] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.342947 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.343911 1584 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.319237 1583 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:49.346529 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.347229 1655 raft_consensus.cc:3037] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:49.347703 4930 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.349093 4904 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.350137 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.350867 1583 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.354730 1654 raft_consensus.cc:1226] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:49.354640 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.355374 1658 raft_consensus.cc:3037] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.355403 4823 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.343134 4907 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.354650 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.357622 4823 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.358017 1735 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260927 23:31:49.355917 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:49.362967 1734 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260927 23:31:49.350889 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.355904 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0803b6723b37453ca7518fb7ced3081d" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.358109 4907 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.367120 1654 raft_consensus.cc:1226] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:49.368223 1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46522 (request call id 1684) took 1370 ms. Trace:
I20260927 23:31:49.368683 1586 rpcz_store.cc:276] 0927 23:31:47.998195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:48.007570 (+ 9375us) service_pool.cc:227] Handling call
0927 23:31:49.368008 (+1360438us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b540007d354. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a732ae 00007f97559b0c27 00007f974f098c04 00007f974f098b92 00007f974f098ac1 00007f974f098a3c 00007f974f097cf2 00007f974ee9994c 00007f974ee98d86 00007f974ee9872f
0927 23:31:49.368153 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1120182528}
I20260927 23:31:49.369123 1702 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.369941 4646 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.370097 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.370411 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.371491 4913 consensus_queue.cc:1046] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:49.372315 1702 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.372556 4913 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:49.372969 4913 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.373052 4687 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:49.371037 1654 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.373863 1584 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260927 23:31:49.374886 1655 raft_consensus.cc:2454] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 9.
I20260927 23:31:49.383725 1584 pending_rounds.cc:76] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 4
I20260927 23:31:49.367673 1731 raft_consensus.cc:1261] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 23:31:49.345227 4557 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.383296 2294 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 5.
I20260927 23:31:49.384012 1584 pending_rounds.cc:97] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260927 23:31:49.390862 1584 raft_consensus.cc:2866] T 0803b6723b37453ca7518fb7ced3081d P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:49.391021 4894 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260927 23:31:49.393748 1551 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
W20260927 23:31:49.317422 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 73392.2us
I20260927 23:31:49.395453 1628 leader_election.cc:302] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:49.397111 4907 raft_consensus.cc:2726] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:49.398609 4904 raft_consensus.cc:482] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.391141 4806 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.399230 4904 raft_consensus.cc:504] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.401965 4904 leader_election.cc:288] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.402386 4806 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Requested vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501)
I20260927 23:31:49.448750 1735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:49.450121 1735 raft_consensus.cc:2379] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:49.403262 4883 leader_election.cc:288] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.406613 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.451460 1661 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:49.451388 4826 raft_consensus.cc:504] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.452554 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.453127 1661 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.455065 4646 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.455708 1552 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.455592 4646 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.457015 4841 consensus_queue.cc:1035] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 23:31:49.457577 4646 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.407097 4862 raft_consensus.cc:3032] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:49.458400 4841 raft_consensus.cc:2781] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:49.388118 1655 raft_consensus.cc:1261] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:49.459125 4841 raft_consensus.cc:482] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.459589 4841 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.460042 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575009df7d846d997a605fb1e38ad31" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.460597 1655 raft_consensus.cc:3037] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:49.461344 1661 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:49.389454 1579 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:49.462177 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.408087 1628 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.462870 1661 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 5.
I20260927 23:31:49.409039 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.463632 4905 raft_consensus.cc:2726] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:49.463896 1657 raft_consensus.cc:2379] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 5: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:49.465086 1657 raft_consensus.cc:1261] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260927 23:31:49.409730 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.466728 1663 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:49.383236 4687 raft_consensus.cc:482] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.467376 4687 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:49.374161 4931 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.410585 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.411094 1660 raft_consensus.cc:1226] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 23:31:49.469944 4931 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.470609 1659 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:49.411536 1658 raft_consensus.cc:2454] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:49.412016 1656 raft_consensus.cc:1261] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Refusing update from remote peer 0fe98f486bab4d0b9c421c145d83f1ce: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 23:31:49.417425 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.382979 4913 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:49.472237 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 64223us
I20260927 23:31:49.466306 1657 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:49.472191 1731 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:49.329413 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.384423 1581 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.420480 1737 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 23:31:49.423484 1629 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:49.483562 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 60186.2us
I20260927 23:31:49.423838 4565 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:49.485703 4687 raft_consensus.cc:504] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.488020 4806 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:31:49.424238 1654 raft_consensus.cc:2454] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:49.425333 1551 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.425455 4733 consensus_queue.cc:1046] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:49.490993 4733 raft_consensus.cc:3032] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:49.491467 4733 raft_consensus.cc:729] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:49.492559 4733 consensus_queue.cc:260] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.425997 4847 consensus_queue.cc:1035] T d0bbde2432f547cbab3588c16fb623cf P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 23:31:49.428406 4793 raft_consensus.cc:504] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.495743 4687 leader_election.cc:288] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.432386 4854 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:49.438510 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 103790us
I20260927 23:31:49.440150 1554 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:49.441494 4722 raft_consensus.cc:979] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: : Instructing follower 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:49.442791 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.497211 2257 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:49.497895 2293 raft_consensus.cc:2354] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
W20260927 23:31:49.498169 4827 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260927 23:31:49.498570 4611 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:49.498970 4611 raft_consensus.cc:482] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.499341 4611 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.494083 4565 raft_consensus.cc:2726] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 23:31:49.497116 1628 leader_election.cc:302] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.501642 2259 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
W20260927 23:31:49.502142 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 76948us
I20260927 23:31:49.503021 1551 leader_election.cc:398] T 9c719671f35640549c1b944112b3fc03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051) with higher term. Message: Invalid argument: T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 6. Current term is 7.
I20260927 23:31:49.503592 4931 raft_consensus.cc:2726] T adf0ac4f0f334ff6862e45d38f191756 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 23:31:49.504156 1579 raft_consensus.cc:1261] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 23:31:49.504225 4733 raft_consensus.cc:3037] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.481772 4913 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.503201 4810 raft_consensus.cc:2781] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 23:31:49.506826 4810 raft_consensus.cc:686] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 9 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:49.507799 4810 consensus_queue.cc:237] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.509029 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.509649 1587 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:49.510069 4913 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.511263 4827 raft_consensus.cc:2726] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:49.511972 4827 consensus_queue.cc:1046] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 23:31:49.513055 4847 raft_consensus.cc:3032] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Stepping down as leader of term 4
I20260927 23:31:49.513365 4847 raft_consensus.cc:729] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
W20260927 23:31:49.514590 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 74554.6us
I20260927 23:31:49.514757 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.515082 4904 raft_consensus.cc:2781] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader election won for term 5
I20260927 23:31:49.443075 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.515848 4904 raft_consensus.cc:686] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:49.443675 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.442214 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.444585 1728 raft_consensus.cc:1261] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 23:31:49.446012 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797da3c81264653acefad127c4e4b2e" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:49.442922 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.446401 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.517632 4722 raft_consensus.cc:1067] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Signalling peer 1b945905616b4989853b0a92bfbd6b98 to start an election
I20260927 23:31:49.407423 4861 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:49.517246 1730 raft_consensus.cc:3037] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.517907 1627 leader_election.cc:302] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.518168 4861 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.518709 4861 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:49.518729 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 98812.6us
I20260927 23:31:49.496034 4793 leader_election.cc:288] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.519721 4854 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:49.522588 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 79137.1us
I20260927 23:31:49.523279 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:49.523535 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.523874 1732 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.523844 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.525549 4854 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.526274 1728 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260927 23:31:49.526633 1728 pending_rounds.cc:76] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Aborting all ops after (but not including) 4
I20260927 23:31:49.526952 1728 pending_rounds.cc:97] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260927 23:31:49.527170 1728 raft_consensus.cc:2866] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:49.527935 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.526525 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.528719 4793 consensus_queue.cc:1035] T 2bac3d0a2c8444f79b96925750c29b42 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 23:31:49.527590 4931 raft_consensus.cc:2781] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:49.447096 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:49.310636 4572 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.533306 1734 raft_consensus.cc:3037] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.533450 4931 raft_consensus.cc:686] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:49.446715 4896 consensus_queue.cc:1035] T fa9f6499247b4a1e9ef1440ac8906b50 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 23:31:49.536650 1703 leader_election.cc:302] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:49.458482 4862 raft_consensus.cc:729] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:49.524279 2259 leader_election.cc:302] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:49.528951 1737 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:49.514197 4847 consensus_queue.cc:260] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.531200 1732 raft_consensus.cc:2454] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:49.531972 1587 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 11.
I20260927 23:31:49.532311 1629 leader_election.cc:398] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
W20260927 23:31:49.535542 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 79898.5us
I20260927 23:31:49.535666 1730 raft_consensus.cc:2454] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:49.494308 1655 raft_consensus.cc:2454] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 9.
I20260927 23:31:49.538264 1581 raft_consensus.cc:2454] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 7.
I20260927 23:31:49.540848 1629 leader_election.cc:401] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260927 23:31:49.543685 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 46636.2us
I20260927 23:31:49.543816 1734 raft_consensus.cc:2454] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 8.
I20260927 23:31:49.541280 1552 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.539047 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.562881 1654 raft_consensus.cc:2454] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 5.
I20260927 23:31:49.553961 1629 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.569761 1629 leader_election.cc:302] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:49.614233 4854 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:49.538010 4862 consensus_queue.cc:260] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:49.641957 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 81548.5us
I20260927 23:31:49.643062 2257 leader_election.cc:302] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
W20260927 23:31:49.643229 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 69415us
I20260927 23:31:49.541811 4905 raft_consensus.cc:2741] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260927 23:31:49.628276 4705 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 23:31:49.630007 4646 consensus_queue.cc:1046] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260927 23:31:49.654919 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 85277.9us
I20260927 23:31:49.632644 4263 consensus_queue.cc:1035] T 18302bf8887a4eb685e0fa76572b5605 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:49.634626 1702 leader_election.cc:302] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.662721 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.639175 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.641146 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:49.641690 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 117467us
I20260927 23:31:49.623487 4823 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:49.539038 4716 raft_consensus.cc:2781] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:49.454138 4826 leader_election.cc:288] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.542534 4637 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.572014 4640 raft_consensus.cc:2726] T f797da3c81264653acefad127c4e4b2e P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509) with higher term. Message: Invalid argument: T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 for earlier term 7. Current term is 8.
I20260927 23:31:49.650804 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:49.649791 4879 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20260927 23:31:49.534459 4931 consensus_queue.cc:237] T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.653383 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.614872 4907 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 1b945905616b4989853b0a92bfbd6b98 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:49.656652 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.543340 4861 raft_consensus.cc:504] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.658566 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.658931 2286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:49.649595 4822 raft_consensus.cc:2781] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
W20260927 23:31:49.733568 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 77915us
I20260927 23:31:49.617548 4854 raft_consensus.cc:482] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.516654 4611 raft_consensus.cc:504] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:49.742939 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 92258.1us
I20260927 23:31:49.615135 4839 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.130s
I20260927 23:31:49.735200 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:49.742874 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 99934.7us
I20260927 23:31:49.657449 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.767441 1587 raft_consensus.cc:2362] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 9. Re-sending same reply.
I20260927 23:31:49.767544 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:49.747045 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "505e560fa0174a5987736c3bb71c7753" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:49.747181 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 98879.9us
I20260927 23:31:49.747822 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.651837 4896 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.105s
I20260927 23:31:49.568822 4687 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260927 23:31:49.754302 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5355e82529343f5b479b2450cefa8ea" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.758064 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.517560 4904 consensus_queue.cc:237] T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:49.753986 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 119484us
I20260927 23:31:49.781203 1657 raft_consensus.cc:1261] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 23:31:49.749011 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.774622 1586 raft_consensus.cc:2379] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:49.775162 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.776871 1627 leader_election.cc:302] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.781024 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.647986 4557 raft_consensus.cc:2726] T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260927 23:31:49.758706 4640 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:49.781116 1728 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:49.757647 4826 raft_consensus.cc:3032] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:49.758520 4883 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.761785 4854 raft_consensus.cc:3037] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.764714 4895 raft_consensus.cc:482] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.782442 1656 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.790877 2292 raft_consensus.cc:2379] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:49.776577 4907 raft_consensus.cc:482] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.793277 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.794243 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "502de1f57afe4b808a200ece197c19e6" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.763866 4822 raft_consensus.cc:686] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:49.769256 4894 raft_consensus.cc:2781] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:49.791530 1588 raft_consensus.cc:2379] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:49.775334 4896 consensus_queue.cc:1225] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Peer 1b945905616b4989853b0a92bfbd6b98 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260927 23:31:49.787204 4930 raft_consensus.cc:2781] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:49.810412 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.767810 4823 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.852507 1551 leader_election.cc:302] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:49.812081 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:49.814675 4901 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.024s sys 0.020s
I20260927 23:31:49.814793 4444 raft_consensus.cc:2781] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:49.810029 4895 raft_consensus.cc:504] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.815415 4799 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260927 23:31:49.816426 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.807960 4894 raft_consensus.cc:482] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.820384 4557 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.860673 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.821338 4822 consensus_queue.cc:237] T e6bcce456c08487491b881518f4f8b46 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.822613 4572 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.826998 4687 raft_consensus.cc:482] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
W20260927 23:31:49.856407 4901 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s user 0.024s sys 0.021s
I20260927 23:31:49.830003 1700 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:49.830433 4907 raft_consensus.cc:504] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.830871 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:49.831174 4937 log.cc:924] Time spent T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47: Append to log took a long time: real 0.214s user 0.001s sys 0.003s
I20260927 23:31:49.832068 1658 raft_consensus.cc:1261] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 23:31:49.831993 1588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.857692 4930 raft_consensus.cc:686] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:49.834028 1660 raft_consensus.cc:1226] T 7fcc903f42c448e1beb437468961e2b0 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 6. Current term is 7. Ops: []
I20260927 23:31:49.834686 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.835474 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.846637 4806 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260927 23:31:49.863644 4894 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.804100 1587 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:49.788431 4640 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:49.797348 4793 raft_consensus.cc:2781] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:49.792394 4826 raft_consensus.cc:729] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:49.856842 2288 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:49.818985 4611 leader_election.cc:288] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.808821 4906 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:49.899789 1657 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 9.
I20260927 23:31:49.900372 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.901109 1656 raft_consensus.cc:3037] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.799415 4883 raft_consensus.cc:504] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.930394 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.930423 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9680f1581344928ab0dd3e14eebea73" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.932099 1663 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.905304 1588 raft_consensus.cc:2379] T c947ede0cb8a48cb90cfb0bcc8ed442b P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 6: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:49.832995 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:49.907037 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 77141.5us
I20260927 23:31:49.907645 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.866480 4733 raft_consensus.cc:2726] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 23:31:49.909610 1661 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.865510 4444 raft_consensus.cc:686] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:49.926296 1585 raft_consensus.cc:2379] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:49.934312 1659 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:49.935665 1700 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:49.936257 1729 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 23:31:49.936772 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4ed89195cf4ee49eeed018ec5039e7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
W20260927 23:31:49.938732 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 147754us
I20260927 23:31:49.930998 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.941763 1658 raft_consensus.cc:2454] T 1c4ed89195cf4ee49eeed018ec5039e7 P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:49.941963 1660 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:49.903051 4557 raft_consensus.cc:504] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.944140 1661 raft_consensus.cc:1226] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:49.945354 4444 consensus_queue.cc:237] T f797da3c81264653acefad127c4e4b2e P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.945966 4895 leader_election.cc:288] T b8ceb00d8a73495dbaab22eda5654033 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.922755 4640 raft_consensus.cc:3037] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:49.946722 4823 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:49.949169 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:49.948002 4861 leader_election.cc:288] T ac10d629e97c4945b1f2fb7ba7c1540e P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:49.922005 4793 raft_consensus.cc:482] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:49.912992 4628 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:49.958642 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:49.959549 1663 raft_consensus.cc:3037] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.963172 1659 raft_consensus.cc:1261] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 23:31:49.963223 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bcce6620eee4240b1d1b454ac3ca6eb" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.964583 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.965355 1587 raft_consensus.cc:2379] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:49.966270 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.966480 1588 raft_consensus.cc:1261] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260927 23:31:49.967303 1583 raft_consensus.cc:1226] T 5bcce6620eee4240b1d1b454ac3ca6eb P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 8. Current term is 9. Ops: []
I20260927 23:31:49.967639 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.968087 1582 raft_consensus.cc:1261] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260927 23:31:49.968436 1582 pending_rounds.cc:76] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 4
I20260927 23:31:49.968324 1581 raft_consensus.cc:2454] T 9629fafc0b4640efa0d437d468e2bdac P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 7.
I20260927 23:31:49.968778 1582 pending_rounds.cc:97] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260927 23:31:49.969066 1582 raft_consensus.cc:2866] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:49.969401 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c7c13b00be47d38145b16ad8a788cc" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.965288 1658 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260927 23:31:49.967324 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:49.950364 4905 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:49.945376 4557 leader_election.cc:288] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.977329 1628 leader_election.cc:302] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:49.922359 4687 raft_consensus.cc:504] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.960870 4793 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:49.978505 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b04c203146534f018c1bdcc2596f0010" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:49.990661 1580 raft_consensus.cc:3037] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:49.954973 4841 raft_consensus.cc:504] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.994616 1584 raft_consensus.cc:2379] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 7: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:49.995250 1700 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.996260 1700 leader_election.cc:302] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:49.973399 4823 raft_consensus.cc:2781] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:49.940551 4906 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:49.976652 4847 raft_consensus.cc:3037] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:50.003741 4807 raft_consensus.cc:2726] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.006633 4841 leader_election.cc:288] T 4912c6b1a48e4d9087c4b72bef319110 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:49.960671 4733 consensus_queue.cc:1046] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260927 23:31:50.007658 4263 raft_consensus.cc:3032] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Stepping down as leader of term 6
I20260927 23:31:50.008004 4263 raft_consensus.cc:729] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:50.001959 4823 raft_consensus.cc:482] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:50.009356 4823 raft_consensus.cc:3037] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.011101 4807 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:50.011631 4687 leader_election.cc:288] T b9fe23b5411b441fbd8792ef9dc4e99b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.012156 4687 consensus_queue.cc:1035] T 9154850d4ac4425dae5bbfde7004d809 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:31:50.014571 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9629fafc0b4640efa0d437d468e2bdac" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.016062 4687 raft_consensus.cc:482] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.016602 4687 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.018685 4687 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.018576 4572 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 23:31:50.008791 4263 consensus_queue.cc:260] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:49.979425 4907 leader_election.cc:288] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.020820 1628 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.027614 1551 leader_election.cc:302] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:50.026733 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 262115us
I20260927 23:31:49.980891 4861 consensus_queue.cc:1035] T 9629fafc0b4640efa0d437d468e2bdac P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260927 23:31:50.002125 4895 raft_consensus.cc:2781] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 23:31:50.038255 1729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:49.978248 4913 consensus_queue.cc:1035] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260927 23:31:50.054702 4263 raft_consensus.cc:3037] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:49.982856 4905 raft_consensus.cc:504] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:50.035727 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 260831us
W20260927 23:31:50.034101 4827 consensus_queue.cc:1159] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2a3317e8da742d9aefe4c251abd4566" responder_term: 6 status { last_received { term: 6 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 6 index: 5 } }
I20260927 23:31:50.064194 1580 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 0fe98f486bab4d0b9c421c145d83f1ce [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 10.
I20260927 23:31:50.078940 4841 raft_consensus.cc:482] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:50.080226 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 312747us
I20260927 23:31:49.978941 4930 consensus_queue.cc:237] T 5d40ddbf07dc427f9a4f51a71f25d985 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.042466 4895 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:50.081043 1663 raft_consensus.cc:2454] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:50.034951 4839 raft_consensus.cc:482] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.081802 2259 leader_election.cc:302] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:50.079650 4841 raft_consensus.cc:504] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:50.098047 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 101582us
I20260927 23:31:50.024602 4826 consensus_queue.cc:260] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.099411 2259 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.088680 4733 raft_consensus.cc:2781] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader election won for term 9
W20260927 23:31:50.104533 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 103492us
I20260927 23:31:50.105321 1628 leader_election.cc:302] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.105564 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.091320 4895 raft_consensus.cc:3037] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.104179 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb9ba54d39e48e08eefe27bddb5fd3b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.091734 4862 raft_consensus.cc:3037] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.106441 2288 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 7.
I20260927 23:31:50.112201 1628 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.118415 1551 leader_election.cc:302] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:50.095633 4883 leader_election.cc:288] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.124490 1628 leader_election.cc:302] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.126070 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:50.126921 1628 leader_election.cc:302] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.104214 4896 raft_consensus.cc:2781] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 FOLLOWER]: Leader election won for term 11
I20260927 23:31:50.109820 4799 raft_consensus.cc:2781] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:50.134152 1703 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:50.135082 1628 leader_election.cc:302] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.135239 1629 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters:
I20260927 23:31:50.140687 1628 leader_election.cc:302] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.143203 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1101ef7672c345ec8f5509edeb86b553" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:50.151366 2292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ddb51781fec46b194174de5f4406def"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
mode: GRACEFUL
new_leader_uuid: "a2a3317e8da742d9aefe4c251abd4566"
from {username='slave'} at 127.0.0.1:51840
I20260927 23:31:50.152122 2292 raft_consensus.cc:595] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 LEADER]: Received request to transfer leadership to TS a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.153085 1657 raft_consensus.cc:3037] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Advancing to term 11
I20260927 23:31:50.153492 1554 leader_election.cc:302] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce; no voters:
I20260927 23:31:50.100389 4806 consensus_queue.cc:1035] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260927 23:31:50.122540 4733 raft_consensus.cc:686] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:50.094638 4839 raft_consensus.cc:504] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } attrs { replace: true } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.205384 4732 consensus_queue.cc:1035] T 8ab559a361d74acca8ee9c195b91d830 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 23:31:50.207768 4841 leader_election.cc:288] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.213718 4839 leader_election.cc:288] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.129642 4807 raft_consensus.cc:2726] T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:50.217485 4807 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:50.218076 4807 raft_consensus.cc:686] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [term 7 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:50.219197 4807 consensus_queue.cc:237] T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.129463 4931 raft_consensus.cc:2726] T 505e560fa0174a5987736c3bb71c7753 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.222574 4931 raft_consensus.cc:2726] T 879c1897331c4c32ade3f828689195c4 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.224339 4905 leader_election.cc:288] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.129642 4557 raft_consensus.cc:2726] T e5355e82529343f5b479b2450cefa8ea P 1b945905616b4989853b0a92bfbd6b98 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 23:31:50.168817 1663 raft_consensus.cc:1261] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 23:31:50.163153 4637 raft_consensus.cc:2726] T 9154850d4ac4425dae5bbfde7004d809 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 23:31:50.152374 4879 raft_consensus.cc:2726] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:50.106041 4854 raft_consensus.cc:504] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.117563 4646 consensus_queue.cc:1046] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260927 23:31:50.240097 1579 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:50.133126 4827 raft_consensus.cc:2741] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260927 23:31:50.166075 4705 raft_consensus.cc:2781] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:50.241228 4705 raft_consensus.cc:482] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:50.241663 4705 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:50.179086 1554 leader_election.cc:302] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.184648 4863 consensus_queue.cc:1035] T 2686f0078ab94fbd99b4744ba43a0980 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260927 23:31:50.152761 4904 raft_consensus.cc:482] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.188822 4906 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.144315 4883 raft_consensus.cc:2726] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:50.198740 4565 consensus_queue.cc:1035] T b8ceb00d8a73495dbaab22eda5654033 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 23:31:50.201110 1661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902"
dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:36962
I20260927 23:31:50.201259 4861 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.230575 1656 raft_consensus.cc:2454] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
I20260927 23:31:50.237751 1660 raft_consensus.cc:2454] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 10.
I20260927 23:31:50.236007 4716 raft_consensus.cc:686] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [term 7 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:50.251857 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74c5e2f6966148308b5ccecb8d8f4313" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:50.277158 1552 leader_election.cc:302] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:50.300036 1659 raft_consensus.cc:3037] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.187222 4733 consensus_queue.cc:237] T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
W20260927 23:31:50.308557 1702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.007s sys 0.061s
W20260927 23:31:50.310155 1702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.510s user 0.007s sys 0.061s
I20260927 23:31:50.281051 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:50.284812 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 105831us
I20260927 23:31:50.238857 4640 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.286679 1661 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:31:50.340795 1661 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.300386 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.237428 4793 raft_consensus.cc:504] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.307013 4861 raft_consensus.cc:504] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.299167 4839 raft_consensus.cc:2726] T 2fb9ba54d39e48e08eefe27bddb5fd3b P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.343993 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.284024 4906 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.363952 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.289146 4827 raft_consensus.cc:482] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.299921 4907 raft_consensus.cc:2741] T b04c203146534f018c1bdcc2596f0010 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260927 23:31:50.350528 4799 raft_consensus.cc:686] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:50.370402 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.372051 4263 raft_consensus.cc:482] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.373402 1663 pending_rounds.cc:76] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Aborting all ops after (but not including) 7
I20260927 23:31:50.368070 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.341118 4841 raft_consensus.cc:2781] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:50.348232 4572 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.340240 4883 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.381456 1663 pending_rounds.cc:97] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260927 23:31:50.409965 1663 raft_consensus.cc:2866] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:50.378010 2286 raft_consensus.cc:1226] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer a2a3317e8da742d9aefe4c251abd4566 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260927 23:31:50.384444 2291 raft_consensus.cc:2379] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 in current term 4: Already voted for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in this term.
I20260927 23:31:50.385236 4793 leader_election.cc:288] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.385316 1579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7fcc903f42c448e1beb437468961e2b0"
dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:47570
I20260927 23:31:50.386013 4716 consensus_queue.cc:237] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.389258 4896 raft_consensus.cc:686] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 11 LEADER]: Becoming Leader. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:50.395375 4572 raft_consensus.cc:504] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.377055 4906 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.392120 4861 leader_election.cc:288] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.390233 4640 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.357741 4854 leader_election.cc:288] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.428776 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f662be9387d148199907fd0e049ba4be" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.430878 1579 raft_consensus.cc:595] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:50.496135 1579 raft_consensus.cc:603] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 23:31:50.287647 4823 raft_consensus.cc:504] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.431958 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4912c6b1a48e4d9087c4b72bef319110" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:50.287477 4904 raft_consensus.cc:504] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.488332 1703 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:50.398423 4841 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:50.397851 4826 raft_consensus.cc:3037] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:50.502986 1731 raft_consensus.cc:2379] T 4912c6b1a48e4d9087c4b72bef319110 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 7: Already voted for candidate a2a3317e8da742d9aefe4c251abd4566 in this term.
I20260927 23:31:50.498124 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bac3d0a2c8444f79b96925750c29b42" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:50.518594 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 89569.1us
I20260927 23:31:50.519045 1730 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:50.428133 4263 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.521209 4705 raft_consensus.cc:504] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.521907 4263 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.523213 4705 leader_election.cc:288] T 07438388789747e68da9846e153a903b P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 election: Requested vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.524771 4896 consensus_queue.cc:237] T 1101ef7672c345ec8f5509edeb86b553 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.495652 4557 raft_consensus.cc:482] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.528214 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d40ddbf07dc427f9a4f51a71f25d985" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:50.556176 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 415510us
I20260927 23:31:50.529935 1702 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:50.556617 1736 raft_consensus.cc:2379] T 5d40ddbf07dc427f9a4f51a71f25d985 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce in current term 8: Already voted for candidate 1b945905616b4989853b0a92bfbd6b98 in this term.
I20260927 23:31:50.381420 4827 raft_consensus.cc:504] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.562245 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
I20260927 23:31:50.558832 1661 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.567492 1661 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.509135 4841 raft_consensus.cc:3037] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:50.475030 4883 raft_consensus.cc:504] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.557160 1628 leader_election.cc:302] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:50.518455 4444 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.024s sys 0.043s
I20260927 23:31:50.574783 1628 leader_election.cc:302] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.576814 1628 leader_election.cc:302] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:50.578181 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 443003us
I20260927 23:31:50.577000 1702 leader_election.cc:302] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
W20260927 23:31:50.565441 4822 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.006s sys 0.013s
I20260927 23:31:50.586972 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566"
W20260927 23:31:50.586578 1737 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.004s sys 0.014s
W20260927 23:31:50.587955 1737 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s user 0.004s sys 0.014s
I20260927 23:31:50.593106 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154850d4ac4425dae5bbfde7004d809" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.603297 1702 leader_election.cc:302] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:50.561787 4705 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.607158 1628 leader_election.cc:302] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.612756 1659 raft_consensus.cc:2454] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
I20260927 23:31:50.575932 4944 consensus_queue.cc:1035] T 0803b6723b37453ca7518fb7ced3081d P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
W20260927 23:31:50.586304 4822 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.006s sys 0.013s
W20260927 23:31:50.575266 4444 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.024s sys 0.043s
I20260927 23:31:50.632447 4930 raft_consensus.cc:2726] T a9680f1581344928ab0dd3e14eebea73 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:50.576057 4687 raft_consensus.cc:2781] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:50.570221 4640 raft_consensus.cc:482] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.641811 1660 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 23:31:50.641646 1629 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:50.642374 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e412df23ce8e4e71bc3c5bc171ca0e85" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:50.648293 1656 raft_consensus.cc:3037] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:50.586745 4931 raft_consensus.cc:2726] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.652858 1657 raft_consensus.cc:2454] T 1101ef7672c345ec8f5509edeb86b553 P 1b945905616b4989853b0a92bfbd6b98 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 11.
I20260927 23:31:50.583617 4905 raft_consensus.cc:2781] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 23:31:50.592011 4863 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.635681 4861 raft_consensus.cc:2726] T 99c7c13b00be47d38145b16ad8a788cc P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:50.572301 4557 raft_consensus.cc:504] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:50.673936 4862 consensus_meta.cc:287] T 505e560fa0174a5987736c3bb71c7753 P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.559s user 0.003s sys 0.003s
I20260927 23:31:50.682401 4953 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:50.682855 4883 leader_election.cc:288] T a191cb6c79334070a9ff2a1a9db5604c P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.658056 4905 raft_consensus.cc:482] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:50.653637 4948 raft_consensus.cc:2781] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 23:31:50.686112 4906 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.658864 4687 raft_consensus.cc:686] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [term 6 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:50.687513 4863 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.690759 4930 raft_consensus.cc:2726] T 5bcce6620eee4240b1d1b454ac3ca6eb P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 23:31:50.693344 4904 leader_election.cc:288] T 61b00bbf7bb549d790e4d375530f3e03 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.696173 4823 leader_election.cc:288] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.605854 4799 consensus_queue.cc:237] T 0be2a211b7754af89c38527623e5d967 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.693168 1656 raft_consensus.cc:2454] T e412df23ce8e4e71bc3c5bc171ca0e85 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 6.
W20260927 23:31:50.698331 1658 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.014s sys 0.026s
I20260927 23:31:50.717814 4905 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Advancing to term 7
W20260927 23:31:50.756249 1658 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.014s sys 0.026s
W20260927 23:31:50.720290 1551 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.007s sys 0.036s
W20260927 23:31:50.728857 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 368401us
I20260927 23:31:50.666530 4931 raft_consensus.cc:482] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.642725 4705 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:50.757472 1551 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.008s sys 0.036s
I20260927 23:31:50.715338 4944 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
W20260927 23:31:50.749608 4894 consensus_meta.cc:287] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.819s user 0.012s sys 0.000s
I20260927 23:31:50.659073 4640 raft_consensus.cc:504] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:50.763461 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 595827us
I20260927 23:31:50.763998 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:50.770169 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee6b5008b764b60a2808b4af785ef8a" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:50.761641 4847 consensus_meta.cc:287] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.736s user 0.011s sys 0.000s
I20260927 23:31:50.779804 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:50.766852 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92d480f46ebe4fafb41af6682200fbb8" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:50.780686 1579 raft_consensus.cc:3037] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:50.657326 4954 consensus_queue.cc:1046] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.116s
I20260927 23:31:50.748392 4948 raft_consensus.cc:686] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [term 10 LEADER]: Becoming Leader. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:50.781529 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:50.786886 1582 raft_consensus.cc:2396] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2a3317e8da742d9aefe4c251abd4566 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:50.780872 1584 raft_consensus.cc:2454] T 92d480f46ebe4fafb41af6682200fbb8 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 9.
I20260927 23:31:50.782305 1586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:50.764842 1552 leader_election.cc:302] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.788074 1586 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 4.
I20260927 23:31:50.783068 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:50.763242 4930 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.763447 4904 raft_consensus.cc:482] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.760275 4637 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.793396 4646 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.795303 1582 raft_consensus.cc:1261] T f575009df7d846d997a605fb1e38ad31 P 0fe98f486bab4d0b9c421c145d83f1ce [term 9 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 23:31:50.800489 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b00bbf7bb549d790e4d375530f3e03" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.801530 4705 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.801291 4646 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.803843 4646 leader_election.cc:288] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.790458 1585 raft_consensus.cc:1261] T 9154850d4ac4425dae5bbfde7004d809 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Refusing update from remote peer a2a3317e8da742d9aefe4c251abd4566: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260927 23:31:50.763943 4931 raft_consensus.cc:504] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.788916 1731 raft_consensus.cc:2454] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
W20260927 23:31:50.819073 1700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 327640us
I20260927 23:31:50.821722 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.822674 1731 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 3.
W20260927 23:31:50.825560 1580 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.002s sys 0.020s
W20260927 23:31:50.826100 1580 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.002s sys 0.020s
I20260927 23:31:50.791118 4628 raft_consensus.cc:2781] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:50.839157 1627 leader_election.cc:302] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:50.843106 1552 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:50.841964 1579 raft_consensus.cc:2454] T ac10d629e97c4945b1f2fb7ba7c1540e P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:50.845329 1551 leader_election.cc:302] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.846562 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08bafa5142a9452db744d03134c85be6" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:50.769868 4944 raft_consensus.cc:504] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.798650 4862 raft_consensus.cc:3032] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 23:31:50.781836 4894 raft_consensus.cc:504] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.824410 4646 consensus_queue.cc:1046] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260927 23:31:50.791852 4961 raft_consensus.cc:2726] T 74c5e2f6966148308b5ccecb8d8f4313 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 23:31:50.838001 4733 raft_consensus.cc:2726] T f662be9387d148199907fd0e049ba4be P 0fe98f486bab4d0b9c421c145d83f1ce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 23:31:50.907913 1728 raft_consensus.cc:2454] T 08bafa5142a9452db744d03134c85be6 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:50.843624 4572 leader_election.cc:288] T ada48ebdd6c1449bb76554223d5569f7 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:50.906714 2286 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "861e779ad2d04547bd9be0bfbc2616c5"
dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
mode: GRACEFUL
new_leader_uuid: "1b945905616b4989853b0a92bfbd6b98"
from {username='slave'} at 127.0.0.1:51840
I20260927 23:31:50.911761 2286 raft_consensus.cc:595] T 861e779ad2d04547bd9be0bfbc2616c5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 10 LEADER]: Received request to transfer leadership to TS 1b945905616b4989853b0a92bfbd6b98
I20260927 23:31:50.849678 4954 raft_consensus.cc:482] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:50.848977 4705 raft_consensus.cc:482] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a2a3317e8da742d9aefe4c251abd4566)
I20260927 23:31:50.804939 4637 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:50.864017 4895 consensus_meta.cc:287] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566: Time spent flushing consensus metadata: real 0.742s user 0.005s sys 0.006s
I20260927 23:31:50.797731 4962 consensus_queue.cc:1035] T 92d480f46ebe4fafb41af6682200fbb8 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260927 23:31:50.854125 4862 raft_consensus.cc:729] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
I20260927 23:31:50.810932 4904 raft_consensus.cc:504] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.919855 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:50.921521 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 50918.1us
I20260927 23:31:50.856326 4827 leader_election.cc:288] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.799285 4930 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.925115 1731 raft_consensus.cc:2396] T 9aa0ad1d9f2349b48e2967755ab789bd P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260927 23:31:50.910794 4263 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:50.928651 4965 raft_consensus.cc:2781] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 23:31:50.928727 4822 consensus_queue.cc:1035] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260927 23:31:50.930735 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 110278us
I20260927 23:31:50.906960 4854 raft_consensus.cc:482] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.917923 4961 raft_consensus.cc:482] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:50.945664 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 309095us
I20260927 23:31:50.912943 4823 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:50.843344 4687 consensus_queue.cc:237] T c947ede0cb8a48cb90cfb0bcc8ed442b P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.914048 4863 leader_election.cc:288] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:50.919168 4565 raft_consensus.cc:2781] T 7fcc903f42c448e1beb437468961e2b0 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election won for term 7
I20260927 23:31:50.965126 1628 leader_election.cc:302] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.977767 1628 leader_election.cc:302] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:50.965958 1551 leader_election.cc:302] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
W20260927 23:31:50.924031 4951 log.cc:924] Time spent T 502de1f57afe4b808a200ece197c19e6 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.319s user 0.000s sys 0.001s
W20260927 23:31:50.962815 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 321219us
I20260927 23:31:50.940467 4263 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:50.989907 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0be2a211b7754af89c38527623e5d967" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.923168 4895 raft_consensus.cc:504] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.984072 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.018445 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:50.966540 4557 leader_election.cc:288] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.967471 4965 raft_consensus.cc:686] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [term 8 LEADER]: Becoming Leader. State: Replica: 1b945905616b4989853b0a92bfbd6b98, State: Running, Role: LEADER
I20260927 23:31:51.036317 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
W20260927 23:31:50.995319 4956 log.cc:924] Time spent T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260927 23:31:50.929725 4954 raft_consensus.cc:504] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.930480 4705 raft_consensus.cc:504] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.037981 1730 raft_consensus.cc:2454] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 6.
I20260927 23:31:51.048192 2292 raft_consensus.cc:1261] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 23:31:51.037477 2291 raft_consensus.cc:3032] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:50.995363 4263 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:51.052783 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879c1897331c4c32ade3f828689195c4" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:51.049170 2291 raft_consensus.cc:729] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ebfcbb9c0574725a9df6ed2bf33ff47, State: Running, Role: LEADER
I20260927 23:31:50.979251 4646 consensus_queue.cc:1035] T 1c4ed89195cf4ee49eeed018ec5039e7 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260927 23:31:51.057252 1584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785f0fd9d6434fb98718143105ca55aa" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:51.057998 1584 raft_consensus.cc:3037] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Advancing to term 7
I20260927 23:31:51.070726 1659 raft_consensus.cc:1226] T 58cf4a7fdcb146b0a364fa00e849e902 P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Rejecting Update request from peer 0ebfcbb9c0574725a9df6ed2bf33ff47 for earlier term 5. Current term is 6. Ops: []
W20260927 23:31:51.070628 1627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 231558us
I20260927 23:31:51.071086 1629 leader_election.cc:302] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:51.083798 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:51.084717 1731 raft_consensus.cc:2454] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebfcbb9c0574725a9df6ed2bf33ff47 in term 3.
I20260927 23:31:50.991148 4883 consensus_queue.cc:1046] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260927 23:31:51.092152 2258 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:51.096463 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a191cb6c79334070a9ff2a1a9db5604c" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.097302 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.138301 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:51.141453 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.047998 4557 raft_consensus.cc:2781] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 23:31:51.054941 4640 leader_election.cc:288] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:50.965972 4854 raft_consensus.cc:504] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.986356 4906 raft_consensus.cc:482] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:50.925473 4948 consensus_queue.cc:237] T b04c203146534f018c1bdcc2596f0010 P a2a3317e8da742d9aefe4c251abd4566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.149664 1579 raft_consensus.cc:2454] T a191cb6c79334070a9ff2a1a9db5604c P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6.
W20260927 23:31:51.150087 1551 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 115768us
I20260927 23:31:51.072844 4827 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:50.995806 4862 consensus_queue.cc:260] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:50.971963 4961 raft_consensus.cc:504] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.156405 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565b68068904064a965cd0a82a20802" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:51.157058 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 223365us
W20260927 23:31:51.156091 2258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 64028.4us
I20260927 23:31:51.099776 2291 consensus_queue.cc:260] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:50.976377 4823 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.084216 4953 raft_consensus.cc:482] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:51.138139 4968 raft_consensus.cc:482] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.164239 1702 leader_election.cc:302] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:51.174721 4983 raft_consensus.cc:2726] T 4912c6b1a48e4d9087c4b72bef319110 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 23:31:51.187212 2291 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:51.168828 4974 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 23:31:51.188961 4974 raft_consensus.cc:482] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:51.189381 4974 raft_consensus.cc:3037] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 4 FOLLOWER]: Advancing to term 5
I20260927 23:31:51.189734 4862 raft_consensus.cc:3037] T 5bcce6620eee4240b1d1b454ac3ca6eb P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Advancing to term 9
I20260927 23:31:51.190213 4954 leader_election.cc:288] T a9680f1581344928ab0dd3e14eebea73 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
W20260927 23:31:51.190689 4950 log.cc:924] Time spent T e850f344b805403cb981ec2cf5272ac2 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.667s user 0.000s sys 0.005s
W20260927 23:31:51.192507 4971 log.cc:924] Time spent T ada48ebdd6c1449bb76554223d5569f7 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.228s user 0.001s sys 0.000s
W20260927 23:31:51.194000 4967 log.cc:924] Time spent T fa9f6499247b4a1e9ef1440ac8906b50 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.226s user 0.000s sys 0.000s
I20260927 23:31:51.195104 4961 leader_election.cc:288] T 18302bf8887a4eb685e0fa76572b5605 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:51.195931 4966 log.cc:924] Time spent T 5ddb51781fec46b194174de5f4406def P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.254s user 0.000s sys 0.001s
I20260927 23:31:51.188968 4969 raft_consensus.cc:2781] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 FOLLOWER]: Leader election won for term 6
I20260927 23:31:51.196882 4969 raft_consensus.cc:686] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [term 6 LEADER]: Becoming Leader. State: Replica: 0fe98f486bab4d0b9c421c145d83f1ce, State: Running, Role: LEADER
I20260927 23:31:51.197083 1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 546) took 1056 ms. Trace:
I20260927 23:31:51.197667 1728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
W20260927 23:31:51.198557 4970 log.cc:924] Time spent T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20260927 23:31:51.197794 4969 consensus_queue.cc:237] T 505e560fa0174a5987736c3bb71c7753 P 0fe98f486bab4d0b9c421c145d83f1ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.199748 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1864) took 1553 ms. Trace:
I20260927 23:31:51.200073 1733 rpcz_store.cc:276] 0927 23:31:49.646286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:49.646778 (+ 492us) service_pool.cc:227] Handling call
0927 23:31:49.728104 (+ 81326us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:50.098106 (+370002us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:50.098150 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:50.105417 (+ 7267us) log.cc:842] Serialized 26 byte log entry
0927 23:31:50.790260 (+684843us) raft_consensus.cc:1631] Marking committed up to 2
0927 23:31:50.790338 (+ 78us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:50.792582 (+ 2244us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.199561 (+406979us) raft_consensus.cc:1671] finished
0927 23:31:51.199580 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.199698 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10123,"raft.queue_time_us":39279,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":41753,"thread_start_us":764987,"threads_started":2,"wal-append.queue_time_us":680830}
I20260927 23:31:51.202123 1702 leader_election.cc:302] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47
I20260927 23:31:51.137696 4931 leader_election.cc:288] T 3a7cf5ede5b947f8a58db3da81e353ef P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:51.151836 4557 raft_consensus.cc:482] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:31:51.137293 4962 consensus_queue.cc:1035] T 61b00bbf7bb549d790e4d375530f3e03 P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260927 23:31:51.168305 4913 raft_consensus.cc:3032] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:51.154599 4894 leader_election.cc:288] T 785f0fd9d6434fb98718143105ca55aa P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 7 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:51.162352 4716 raft_consensus.cc:482] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:51.216611 1655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46200 (request call id 1751) took 1570 ms. Trace:
W20260927 23:31:51.167179 1628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 189471us
I20260927 23:31:51.197465 1732 rpcz_store.cc:276] 0927 23:31:50.140998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:50.141627 (+ 629us) service_pool.cc:227] Handling call
0927 23:31:50.190874 (+ 49247us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:50.191360 (+ 486us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:50.191401 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:50.191989 (+ 588us) log.cc:842] Serialized 26 byte log entry
0927 23:31:50.577870 (+385881us) raft_consensus.cc:1631] Marking committed up to 7
0927 23:31:50.577921 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:50.578007 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.196882 (+618875us) raft_consensus.cc:1671] finished
0927 23:31:51.196904 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.197029 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1418,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":110,"thread_start_us":385398,"threads_started":1,"wal-append.queue_time_us":568456}
I20260927 23:31:51.247566 1552 leader_election.cc:302] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:51.217180 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 475) took 1083 ms. Trace:
I20260927 23:31:51.240345 1731 raft_consensus.cc:2454] T 3565b68068904064a965cd0a82a20802 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 8.
I20260927 23:31:51.203179 1728 raft_consensus.cc:2454] T 3a7cf5ede5b947f8a58db3da81e353ef P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:51.243490 1702 leader_election.cc:302] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2a3317e8da742d9aefe4c251abd4566; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:51.249869 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:51.212371 4954 raft_consensus.cc:2781] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 23:31:51.157956 4444 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:51.254700 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 183682us
W20260927 23:31:51.256368 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 68766.2us
I20260927 23:31:51.215442 4557 raft_consensus.cc:3037] T 9629fafc0b4640efa0d437d468e2bdac P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:51.260010 1629 leader_election.cc:302] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98; no voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:51.241201 1655 rpcz_store.cc:276] 0927 23:31:49.640712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:49.641289 (+ 577us) service_pool.cc:227] Handling call
0927 23:31:49.649928 (+ 8639us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:49.756831 (+106903us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:49.756864 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:49.770279 (+ 13415us) log.cc:842] Serialized 26 byte log entry
0927 23:31:50.127716 (+357437us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:50.127752 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:50.127813 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.190142 (+1062329us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b44000232cc. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f974d98bf11 00007f974da59a95 00007f974da5a4b2 00007f9750702511 00007f97506fde1a 00007f97506f1bee 00007f97506f0f36 00007f97506f0b58 00007f97506fa551 00007f97506fa509 00007f97506fa499
0927 23:31:51.201323 (+ 11181us) raft_consensus.cc:1671] finished
0927 23:31:51.201342 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.211131 (+ 9789us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1552,"raft.queue_time_us":79529,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7883,"spinlock_wait_cycles":9993472,"thread_start_us":328159,"threads_started":1,"wal-append.queue_time_us":747997}
I20260927 23:31:51.257763 1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38812 (request call id 1736) took 1473 ms. Trace:
I20260927 23:31:51.270587 1580 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56820 (request call id 569) took 1155 ms. Trace:
W20260927 23:31:51.271140 4979 log.cc:924] Time spent T 07438388789747e68da9846e153a903b P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260927 23:31:51.272078 4952 log.cc:924] Time spent T 74c5e2f6966148308b5ccecb8d8f4313 P 0fe98f486bab4d0b9c421c145d83f1ce: Append to log took a long time: real 0.626s user 0.000s sys 0.001s
I20260927 23:31:51.222353 4913 raft_consensus.cc:729] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a2a3317e8da742d9aefe4c251abd4566, State: Running, Role: LEADER
W20260927 23:31:51.285564 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 54972.2us
I20260927 23:31:51.270980 1580 rpcz_store.cc:276] 0927 23:31:50.115668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:50.120198 (+ 4530us) service_pool.cc:227] Handling call
0927 23:31:50.120858 (+ 660us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:50.224445 (+103587us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:50.224489 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:50.225720 (+ 1231us) log.cc:842] Serialized 480 byte log entry
0927 23:31:50.832862 (+607142us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:50.832930 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:51.270089 (+437159us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400467654. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:51.270145 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.270197 (+ 52us) raft_consensus.cc:1671] finished
0927 23:31:51.270214 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.270487 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":237254016,"thread_start_us":606631,"threads_started":1,"wal-append.queue_time_us":812370}
I20260927 23:31:51.248714 1662 rpcz_store.cc:276] 0927 23:31:50.132291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:50.227520 (+ 95229us) service_pool.cc:227] Handling call
0927 23:31:50.276732 (+ 49212us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:50.277114 (+ 382us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:50.277141 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:50.290647 (+ 13506us) log.cc:842] Serialized 26 byte log entry
0927 23:31:50.739135 (+448488us) raft_consensus.cc:1631] Marking committed up to 9
0927 23:31:50.997384 (+258249us) log.cc:842] Serialized 14 byte log entry
0927 23:31:51.211032 (+213648us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:51.212252 (+ 1220us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.212931 (+ 679us) raft_consensus.cc:1671] finished
0927 23:31:51.213238 (+ 307us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.215784 (+ 2546us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":477587,"raft.queue_time_us":193268,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":15501,"thread_start_us":811226,"threads_started":3,"wal-append.queue_time_us":611125}
I20260927 23:31:51.300457 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07438388789747e68da9846e153a903b" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47"
I20260927 23:31:51.301146 2292 raft_consensus.cc:3037] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Advancing to term 8
I20260927 23:31:51.175377 4947 consensus_queue.cc:1035] T f575009df7d846d997a605fb1e38ad31 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20260927 23:31:51.233180 4637 leader_election.cc:288] T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:51.305192 1732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2686f0078ab94fbd99b4744ba43a0980" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:51.322969 1731 raft_consensus.cc:1261] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:51.252357 4931 raft_consensus.cc:482] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.771755 4883 raft_consensus.cc:482] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.771649 4931 raft_consensus.cc:504] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.772475 4883 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.734154 4572 raft_consensus.cc:482] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:51.172372 4906 raft_consensus.cc:504] T 87786104826046e7852edc590e194f5d P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.774279 4572 raft_consensus.cc:504] T 2686f0078ab94fbd99b4744ba43a0980 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.265556 1737 rpcz_store.cc:276] 0927 23:31:49.784154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:49.784293 (+ 139us) service_pool.cc:227] Handling call
0927 23:31:49.798750 (+ 14457us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:49.939275 (+140525us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:49.939305 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:49.955893 (+ 16588us) log.cc:842] Serialized 26 byte log entry
0927 23:31:50.588568 (+632675us) raft_consensus.cc:1631] Marking committed up to 4
0927 23:31:50.588621 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:50.730761 (+142140us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400147b54. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:50.730825 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.243963 (+513138us) raft_consensus.cc:1671] finished
0927 23:31:51.243987 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.257704 (+ 13717us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5109,"raft.queue_time_us":93,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":11749,"spinlock_wait_cycles":303959424,"thread_start_us":594142,"threads_started":1,"wal-append.queue_time_us":780438}
I20260927 23:31:51.775908 4994 raft_consensus.cc:482] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0ebfcbb9c0574725a9df6ed2bf33ff47)
I20260927 23:31:51.186481 4827 raft_consensus.cc:504] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.776672 4994 raft_consensus.cc:504] T e382c82189e743b4ace885d63e5e9560 P a2a3317e8da742d9aefe4c251abd4566 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.262988 4954 raft_consensus.cc:482] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1b945905616b4989853b0a92bfbd6b98)
I20260927 23:31:51.263772 4904 leader_election.cc:288] T fa9f6499247b4a1e9ef1440ac8906b50 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:51.262348 4930 leader_election.cc:288] T 8c92025b82e6497ba49fb88677061e6e P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:51.205942 4968 raft_consensus.cc:504] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.253599 4716 raft_consensus.cc:504] T 2b90254fab234e29ba132c165c446490 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.214687 4953 raft_consensus.cc:504] T 861e779ad2d04547bd9be0bfbc2616c5 P a2a3317e8da742d9aefe4c251abd4566 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.205363 4965 consensus_queue.cc:237] T 9aa0ad1d9f2349b48e2967755ab789bd P 1b945905616b4989853b0a92bfbd6b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.275192 4444 raft_consensus.cc:504] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.343525 1581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cf4a7fdcb146b0a364fa00e849e902" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:51.343740 2293 raft_consensus.cc:1226] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:51.484071 1588 raft_consensus.cc:1261] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260927 23:31:51.452768 4894 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:51.430773 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ceb00d8a73495dbaab22eda5654033" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:51.795207 1655 raft_consensus.cc:2454] T b8ceb00d8a73495dbaab22eda5654033 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
I20260927 23:31:51.468360 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18302bf8887a4eb685e0fa76572b5605" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.429353 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c719671f35640549c1b944112b3fc03" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:51.514106 4995 raft_consensus.cc:482] T 08bafa5142a9452db744d03134c85be6 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:51.733817 4969 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.001s sys 0.018s
I20260927 23:31:51.774163 4931 leader_election.cc:288] T d0bbde2432f547cbab3588c16fb623cf P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:51.490329 4913 consensus_queue.cc:260] T e412df23ce8e4e71bc3c5bc171ca0e85 P a2a3317e8da742d9aefe4c251abd4566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
W20260927 23:31:51.476655 4991 log.cc:924] Time spent T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20260927 23:31:51.536554 1627 leader_election.cc:302] T 3565b68068904064a965cd0a82a20802 P 1b945905616b4989853b0a92bfbd6b98 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b945905616b4989853b0a92bfbd6b98, a2a3317e8da742d9aefe4c251abd4566; no voters:
I20260927 23:31:51.378568 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa0ad1d9f2349b48e2967755ab789bd" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce"
I20260927 23:31:51.826145 1583 raft_consensus.cc:2379] T 9aa0ad1d9f2349b48e2967755ab789bd P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1b945905616b4989853b0a92bfbd6b98 in current term 8: Already voted for candidate 0fe98f486bab4d0b9c421c145d83f1ce in this term.
I20260927 23:31:51.521951 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa9f6499247b4a1e9ef1440ac8906b50" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
I20260927 23:31:51.828191 1654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36962 (request call id 481) took 1187 ms. Trace:
I20260927 23:31:51.778307 4827 leader_election.cc:288] T e6bcce456c08487491b881518f4f8b46 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:51.828759 1654 rpcz_store.cc:276] 0927 23:31:50.640763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:50.653066 (+ 12303us) service_pool.cc:227] Handling call
0927 23:31:50.668503 (+ 15437us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 23:31:50.668826 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:50.668852 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 23:31:50.669674 (+ 822us) log.cc:842] Serialized 480 byte log entry
0927 23:31:50.967404 (+297730us) raft_consensus.cc:1631] Marking committed up to 5
0927 23:31:50.967439 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:50.967499 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 23:31:51.827947 (+860448us) raft_consensus.cc:1671] finished
0927 23:31:51.827974 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:51.828125 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":297398,"threads_started":1,"wal-append.queue_time_us":1156194}
I20260927 23:31:51.508304 1730 raft_consensus.cc:1226] T 879c1897331c4c32ade3f828689195c4 P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Rejecting Update request from peer 0fe98f486bab4d0b9c421c145d83f1ce for earlier term 5. Current term is 6. Ops: []
I20260927 23:31:51.580152 2292 raft_consensus.cc:2454] T 07438388789747e68da9846e153a903b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 8.
I20260927 23:31:51.521845 2257 leader_election.cc:302] T 74c5e2f6966148308b5ccecb8d8f4313 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: 0fe98f486bab4d0b9c421c145d83f1ce
I20260927 23:31:51.284906 4646 raft_consensus.cc:482] T 0be2a211b7754af89c38527623e5d967 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:51.496223 1629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 236290us
I20260927 23:31:51.483100 4879 raft_consensus.cc:482] T a191cb6c79334070a9ff2a1a9db5604c P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.383373 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:51.418020 4854 leader_election.cc:288] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), 0ebfcbb9c0574725a9df6ed2bf33ff47 (127.0.99.4:45509)
I20260927 23:31:51.683008 1659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac10d629e97c4945b1f2fb7ba7c1540e" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
I20260927 23:31:51.519740 4895 leader_election.cc:288] T cee6b5008b764b60a2808b4af785ef8a P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 6 election: Requested vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:51.807680 4894 raft_consensus.cc:504] T 8c92025b82e6497ba49fb88677061e6e P a2a3317e8da742d9aefe4c251abd4566 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } }
I20260927 23:31:51.468398 4962 raft_consensus.cc:482] T 5ddb51781fec46b194174de5f4406def P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.500705 4628 raft_consensus.cc:482] T 1c4ed89195cf4ee49eeed018ec5039e7 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0fe98f486bab4d0b9c421c145d83f1ce)
I20260927 23:31:51.498104 4998 raft_consensus.cc:2726] T 2bac3d0a2c8444f79b96925750c29b42 P 1b945905616b4989853b0a92bfbd6b98 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260927 23:31:51.515278 4841 consensus_meta.cc:287] T 3a7cf5ede5b947f8a58db3da81e353ef P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.932s user 0.012s sys 0.000s
W20260927 23:31:51.502498 4973 log.cc:924] Time spent T 2686f0078ab94fbd99b4744ba43a0980 P 1b945905616b4989853b0a92bfbd6b98: Append to log took a long time: real 0.327s user 0.001s sys 0.000s
I20260927 23:31:51.520977 4963 raft_consensus.cc:3032] T 879c1897331c4c32ade3f828689195c4 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 LEADER]: Stepping down as leader of term 5
I20260927 23:31:51.510773 4611 raft_consensus.cc:979] T 5ddb51781fec46b194174de5f4406def P 0ebfcbb9c0574725a9df6ed2bf33ff47: : Instructing follower a2a3317e8da742d9aefe4c251abd4566 to start an election
I20260927 23:31:51.502625 4823 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051)
I20260927 23:31:51.505523 4986 raft_consensus.cc:2726] T 0be2a211b7754af89c38527623e5d967 P a2a3317e8da742d9aefe4c251abd4566 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:51.442236 4705 leader_election.cc:288] T e850f344b805403cb981ec2cf5272ac2 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:51.474646 4807 raft_consensus.cc:2726] T 9c719671f35640549c1b944112b3fc03 P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 23:31:51.750357 1728 raft_consensus.cc:1261] T f797da3c81264653acefad127c4e4b2e P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Refusing update from remote peer 0ebfcbb9c0574725a9df6ed2bf33ff47: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 23:31:51.325868 4640 raft_consensus.cc:482] T ada48ebdd6c1449bb76554223d5569f7 P 1b945905616b4989853b0a92bfbd6b98 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.384399 4861 raft_consensus.cc:2781] T d069fe150057489e8a595e9fe7c32de5 P a2a3317e8da742d9aefe4c251abd4566 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 23:31:51.501948 4978 log.cc:924] Time spent T 8ab559a361d74acca8ee9c195b91d830 P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.242s user 0.000s sys 0.000s
I20260927 23:31:51.330638 4839 raft_consensus.cc:482] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 23:31:51.757951 1584 consensus_meta.cc:287] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce: Time spent flushing consensus metadata: real 0.668s user 0.009s sys 0.001s
I20260927 23:31:51.761129 1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38820 (request call id 1870) took 1462 ms. Trace:
I20260927 23:31:51.751881 4732 raft_consensus.cc:2781] T 08bafa5142a9452db744d03134c85be6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260927 23:31:51.668864 4565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.034s sys 0.103s
I20260927 23:31:51.770849 5001 raft_consensus.cc:482] T 8c92025b82e6497ba49fb88677061e6e P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.407752 1580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7cf5ede5b947f8a58db3da81e353ef" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
W20260927 23:31:51.513670 4987 log.cc:924] Time spent T d58c5d7cee3a4fa7b7b7d55bf8cd213b P a2a3317e8da742d9aefe4c251abd4566: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260927 23:31:51.340183 4637 raft_consensus.cc:482] T 0803b6723b37453ca7518fb7ced3081d P 1b945905616b4989853b0a92bfbd6b98 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:31:51.784828 1581 raft_consensus.cc:3037] T 58cf4a7fdcb146b0a364fa00e849e902 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Advancing to term 6
I20260927 23:31:51.785108 1588 pending_rounds.cc:76] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 4
I20260927 23:31:51.778609 4954 raft_consensus.cc:3037] T 92d480f46ebe4fafb41af6682200fbb8 P a2a3317e8da742d9aefe4c251abd4566 [term 9 FOLLOWER]: Advancing to term 10
I20260927 23:31:51.817634 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c947ede0cb8a48cb90cfb0bcc8ed442b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "1b945905616b4989853b0a92bfbd6b98"
W20260927 23:31:51.828630 2291 consensus_meta.cc:287] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47: Time spent flushing consensus metadata: real 0.640s user 0.007s sys 0.000s
I20260927 23:31:51.344699 1585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab559a361d74acca8ee9c195b91d830" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:51.780936 4968 leader_election.cc:288] T adf0ac4f0f334ff6862e45d38f191756 P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1b945905616b4989853b0a92bfbd6b98 (127.0.99.2:42051), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
W20260927 23:31:51.814648 4969 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.001s sys 0.019s
I20260927 23:31:51.868166 1661 raft_consensus.cc:2454] T 8ab559a361d74acca8ee9c195b91d830 P 1b945905616b4989853b0a92bfbd6b98 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 10.
I20260927 23:31:51.512882 4974 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.526103 4263 raft_consensus.cc:504] T d069fe150057489e8a595e9fe7c32de5 P 0fe98f486bab4d0b9c421c145d83f1ce [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "a2a3317e8da742d9aefe4c251abd4566" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 35731 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } }
I20260927 23:31:51.882545 4444 leader_election.cc:288] T d58c5d7cee3a4fa7b7b7d55bf8cd213b P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0fe98f486bab4d0b9c421c145d83f1ce (127.0.99.1:45501), a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3:35731)
I20260927 23:31:51.417448 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d069fe150057489e8a595e9fe7c32de5" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" is_pre_election: true
W20260927 23:31:51.872845 2259 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.002s sys 0.016s
I20260927 23:31:51.501873 4943 raft_consensus.cc:2932] T f662be9387d148199907fd0e049ba4be P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER a2a3317e8da742d9aefe4c251abd4566 (127.0.99.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45501 } } peers { permanent_uuid: "1b945905616b4989853b0a92bfbd6b98" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 42051 } } peers { permanent_uuid: "0ebfcbb9c0574725a9df6ed2bf33ff47" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 45509 } attrs { promote: false } } }
W20260927 23:31:51.887312 2257 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.2:42051, user_credentials={real_user=slave}} blocked reactor thread for 365543us
W20260927 23:31:52.278980 2259 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.947s user 0.003s sys 0.020s
I20260927 23:31:52.234158 1552 leader_election.cc:302] T cee6b5008b764b60a2808b4af785ef8a P 0fe98f486bab4d0b9c421c145d83f1ce [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fe98f486bab4d0b9c421c145d83f1ce, a2a3317e8da742d9aefe4c251abd4566; no voters:
W20260927 23:31:52.279486 2259 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.1:45501, user_credentials={real_user=slave}} blocked reactor thread for 947626us
I20260927 23:31:52.275202 1654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada48ebdd6c1449bb76554223d5569f7" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:52.282058 1584 raft_consensus.cc:2454] T 785f0fd9d6434fb98718143105ca55aa P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 7.
W20260927 23:31:51.938587 1734 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.084s sys 0.249s
W20260927 23:31:52.283243 1734 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.121s user 0.158s sys 0.376s
I20260927 23:31:52.280635 1655 raft_consensus.cc:2362] T c947ede0cb8a48cb90cfb0bcc8ed442b P 1b945905616b4989853b0a92bfbd6b98 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a2a3317e8da742d9aefe4c251abd4566 in term 6. Re-sending same reply.
I20260927 23:31:52.043761 1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 550) took 1055 ms. Trace:
I20260927 23:31:52.009207 1583 raft_consensus.cc:1261] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: Refusing update from remote peer 1b945905616b4989853b0a92bfbd6b98: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260927 23:31:51.888046 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e850f344b805403cb981ec2cf5272ac2" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:52.285231 1583 pending_rounds.cc:76] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting all ops after (but not including) 4
I20260927 23:31:52.285579 1583 pending_rounds.cc:97] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260927 23:31:52.281411 2257 leader_election.cc:302] T e412df23ce8e4e71bc3c5bc171ca0e85 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebfcbb9c0574725a9df6ed2bf33ff47, 1b945905616b4989853b0a92bfbd6b98; no voters: a2a3317e8da742d9aefe4c251abd4566
I20260927 23:31:51.882802 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bcce456c08487491b881518f4f8b46" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:52.287217 2291 raft_consensus.cc:2454] T 58cf4a7fdcb146b0a364fa00e849e902 P 0ebfcbb9c0574725a9df6ed2bf33ff47 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b945905616b4989853b0a92bfbd6b98 in term 6.
I20260927 23:31:51.887118 1731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf0ac4f0f334ff6862e45d38f191756" candidate_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a2a3317e8da742d9aefe4c251abd4566" is_pre_election: true
I20260927 23:31:52.277734 1582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c92025b82e6497ba49fb88677061e6e" candidate_uuid: "1b945905616b4989853b0a92bfbd6b98" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0fe98f486bab4d0b9c421c145d83f1ce" is_pre_election: true
I20260927 23:31:52.284843 1733 rpcz_store.cc:276] 0927 23:31:50.988243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 23:31:51.212874 (+224631us) service_pool.cc:227] Handling call
0927 23:31:51.213178 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 23:31:51.776004 (+562826us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 23:31:51.776048 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 23:31:51.776126 (+ 78us) raft_consensus.cc:1631] Marking committed up to 8
0927 23:31:51.776170 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 23:31:52.043417 (+267247us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54002be154. stack: 00007f974db15033 00007f974db14f30 00007f974d473534 00007f975717f8b4 00007f975717f849 00007f97509f4471 00007f9750a6ff51 00007f9750a6e99e 00007f97559b02e9 00007f974f096684 00007f974f096612 00007f974f096541 00007f974f0964bc 00007f974f095772 00007f974ee9994c 00007f974ee98d86
0927 23:31:52.043491 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 23:31:52.043681 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1293876992}
I20260927 23:31:52.288748 1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60802 (request call id 555) took 1075 ms. Trace:
I20260927 23:31:52.288117 1731 raft_consensus.cc:2454] T adf0ac4f0f334ff6862e45d38f191756 P a2a3317e8da742d9aefe4c251abd4566 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fe98f486bab4d0b9c421c145d83f1ce in term 4.
I20260927 23:31:52.283838 1588 pending_rounds.cc:97] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260927 23:31:52.289783 1656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe23b5411b441fbd8792ef9dc4e99b" candidate_uuid: "a2a3317e8da742d9aefe4c251abd4566" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1b945905616b4989853b0a92bfbd6b98" is_pre_election: true
I20260927 23:31:52.290125 1588 raft_consensus.cc:2866] T 502de1f57afe4b808a200ece197c19e6 P 0fe98f486bab4d0b9c421c145d83f1ce [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 23:31:51.888746 1554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.99.4:45509, user_credentials={real_user=slave}} blocked reactor thread for 224777us
I20260927 23:31:52.286262 1583 raft_consensus.cc:2866] T 5d40ddbf07dc427f9a4f51a71f25d985 P 0fe98f486bab4d0b9c421c145d83f1ce [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 23:31:52.298095 1737 raft_consensus.cc:2396] T e6bcce456c08487491b881518f4f8b46 P a2a3317e8da742d9aefe4c251abd4566 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0fe98f486bab4d0b9c421c145d83f1ce for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 23:31:52.307353 1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46518 (request call id 1636) took 1006 ms. Trace:
W20260927 23:31:52.310969 1552 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.99.3:35731, user_credentials={real_user=slave}} blocked reactor thread for 76956.4us
I20260927 23:31:52.311484 2291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48306 (request call id 550) took 1270 ms. Trace:
W20260927 23:31:52.085160 1702 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.014s sys 0.028s
W20260927 23:31:52.313082 1702 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.015s sys 0.029s
W20260927 23:31:52.313741 1702 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote ->